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Test  &  Evaluation  Management  Guide 
Foreword 

This  6th  edition  of  the  Test  and  Evaluation  Management  Guide  (TEMG)  was  updated  with  the 
cooperation  and  support  of  the  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and 
Evaluation  (DASD(DT&E)),  the  Director  of  Operational  Test  and  Evaluation  (DOT&E),  the  President  of  the 
Defense  Acquisition  University  (DAU),  and  the  DoD  Component  test  and  evaluation  (T&E) 
representatives.  The  TEMG  is  one  of  many  technical  management  educational  guides  developed  for  use 
by  DAU.  Although  some  Service-specific  processes  are  included  as  illustrative  examples,  the  TEMG,  like 
all  DAU  products,  is  written  primarily  from  a  Department  of  Defense  (DoD)  perspective,  i.e.,  non-Service 
specific. 

The  TEMG  is  intended  primarily  for  use  in  courses  at  DAU  and  secondarily  as  a  generic  desk  reference 
for  program  and  project  management,  and  T&E  personnel.  It  is  written  for  current  and  potential 
acquisition  management  personnel  and  assumes  some  familiarity  with  basic  terms,  definitions,  and 
processes  as  employed  by  the  DoD  acquisition  process.  The  TEMG  is  designed  to  assist  Government  and 
industry  personnel  in  executing  their  management  responsibilities  relative  to  the  T&E  support  of 
defense  systems  and  facilitate  learning  during  DAU  coursework. 

The  objective  of  a  well-managed  T&E  program  is  to  provide  timely  and  accurate  information  to  decision 
makers  and  program  managers  (PMs).  The  TEMG  was  developed  to  assist  the  acquisition  community  in 
obtaining  a  better  understanding  of  who  the  decision  makers  are  and  determining  how  and  when  to 
plan  T&E  events  so  that  they  are  efficient  and  effective. 

Although  the  TEMG  includes  some  references  to  DoD  policies,  it  is  not  a  policy  document  and  should  not 
be  viewed  as  such.  The  latest  versions  of  the  DoD  5000  series,  as  well  as  the  Defense  Acquisition 
Guidebook  (DAG)  (Chapter  9,  Test  and  Evaluation),  should  be  consulted  for  specific  policies  and  DoD 
recommended  practices. 

References  to  Military  Handbooks  (MIL-HDBKs)  appear  throughout  the  TEMG:  however,  these 
handbooks  are  for  guidance  only  and  cannot  be  cited  as  a  requirement. 

For  additional  learning  materials  about  DoD  T&E,  consult  the  DAU  Website  (  www.dau.mil ),  for 
reviewing  T&E  continuous  learning  modules  and  accessing  T&E  courses,  and  go  to  the  DAU  Acquisition 
Community  Connection,  T&E  Community  of  Practice  (CoP)  available  at  https://acc.dau. mil/t&e  . 

Comments,  updates,  changes,  and/or  comments  to  this  guide  should  be  provided  to  Office  of  the 
DASD(DT&E)  T&E  Competency  and  Development  Directorate  and  to  DAU. 
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Test  &  Evaluation  Management  Guide 
Chapter  1 

1.1  Introduction 

This  chapter  provides  an  introduction  to  the  policy  and  organizations  that  govern  the  conduct  of  T&E 
activities  within  the  DoD  and  discusses  congressional  legislation  of  T&E  activities  for  compliance  by  DoD. 
It  outlines  the  responsibilities  of  DoD  test  organizations  at  the  Office  of  the  Secretary  of  Defense  (OSD) 
and  Service  levels  and  describes  related  T&E  policy. 

1.2  Congress 

Congress  requires  the  DoD  to  provide  the  following  reports  that  include  information  on  T&E: 

•  Selected  Acquisition  Report  (SAR).  This  report  consists  of  cost,  schedule,  and  performance  data. 
The  SAR  describes  Acquisition  Category  (ACAT)  I  system  characteristics  required  and  outlines 
significant  progress  and  problems  encountered.  It  lists  tests  completed  and  issues  identified 
during  testing.  The  program  manager  (PM)  uses  the  Consolidated  Acquisition  Reporting  System 
software  to  prepare  the  SAR. 

•  DOT&E  Annual  Report.  This  report  is  provided  by  the  DOT&E  to  the  Secretary  of  Defense 
(SecDef)  and  the  committees  on  Armed  Services,  National  Security,  and  Appropriations.  The 
report  provides  a  narrative  and  resource  summary  of  all  operational  test  and  evaluation  (OT&E) 
and  related  issues,  initiatives,  other  interest  areas,  activities,  and  assessments  in  the  previous 
fiscal  year.  When  oversight  of  live-fire  testing  (LET)  was  moved  to  DOT&E,  this  issue  was  added 
to  the  report. 

•  Beyond  Low-Rate  Initial  Production  (BLRIP)  Report.  Before  proceeding  to  BLRIP  for  each  major 
defense  acquisition  program  (MDAP),  the  DOT&E  must  report  to  the  SecDef,  Deputy  Secretary 
of  Defense,  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD(AT&L)), 
Secretaries  of  the  Military  Departments,  and  Congress.  This  report  addresses  the  adequacy  of 
the  Service  initial  operational  test  &  evaluation  (lOT&E)  and  whether  the  T&E  results  confirm 
that  the  tested  item  or  component  is  effective,  suitable,  and  survivable  for  combat.  When 
oversight  of  live-fire  test  &  evaluation  (LFT&E)  was  moved  to  the  DOT&E,  the  LFT  Report  was 
added  to  the  BLRIP  report  content. 

•  Foreign  Comparative  Testing  (FCT)  Report.  The  USD(AT&L)  should  notify  Congress  a  minimum 
of  30  days  prior  to  the  commitment  of  funds  for  initiation  of  new  FCT  evaluations  of  equipment 
produced  by  select  allied  and  friendly  foreign  countries. 

•  Joint  DASD(DT&E)  and  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering 
(DASD(SE))  Annual  Report.  This  report  is  required  by  statute  to  be  provided  to  the  committees 
on  Armed  Services  and  Appropriations.  The  joint  report  includes  the  significant  Developmental 
Test  and  Evaluation  (DT&E)  and  systems  engineering  (SE)  activities  for  the  Department's  MDAPs, 
major  automated  information  systems  (MAIS),  and  special  interest  programs.  The  report 
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evaluates  the  progress  of  weapon  systems'  performance  for  programs  designated  for  OSD  T&E 
oversight. 

1.3  OSD  Oversight  Structure 

The  DoD  organization  for  the  oversight  of  T&E  is  illustrated  in  Figure  1-1.  For  the  USD(AT&L),  DT&E 
oversight  is  performed  by  the  DASD(DT&E),  within  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Research  and  Engineering  (ASD(R&E)).  The  DOT&E  provides  OT&E  oversight  for  the  SecDef.  The 
management  of  MDAPs  in  OSD  is  performed  by  the  Defense  Acquisition  Executive  (DAE),  who  uses  the 
Defense  Acquisition  Board  (DAB)  and  Overarching  Integrated  Product  Teams  (OIPTs)  to  process 
information  for  decisions. 
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Figure  1-1  DoD  Test  and  Evaluation  Organizations 


1.3.1  DAE 


The  DAE  position,  established  in  September  1986,  is  held  by  the  USD(AT&L).  As  the  DAE,  the  USD(AT&L) 
uses  the  DAB  and  its  OIPTs  to  provide  the  senior-level  decision  process  for  the  acquisition  of  weapon 
systems.  DAE  responsibilities  include  establishing  policies  for  acquisition  (including  procurement. 
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research  and  development  (R&D),  logistics,  developmental  testing  (DT),  and  contracts  administration) 
for  all  elements  of  DoD.  The  DAE's  charter  includes  the  authority  over  the  Services  and  Defense  Agencies 
on  policy,  procedure,  and  execution  of  the  weapon  systems  acquisition  process. 

1.3.2  DAB 

The  DAB  is  the  senior  advisory  board  for  the  DoD  acquisition  system.  The  board  includes  the  Vice  Chair 
of  the  Joint  Chiefs,  the  service  Secretaries,  and  a  number  of  Undersecretaries  of  Defense.  Members  of 
the  DAB  are  responsible  for  approving  the  MDAPs  and  serve  as  the  most  important  executive  review  of 
the  most  expensive  acquisition  projects  in  the  DoD.  The  DAB  is  also  the  principal  review  forum  enabling 
the  USD(AT&L)  to  fulfill  10  USC  Chapter  144  responsibilities  concerning  ACAT  ID  MDAPs. 

The  USD  (AT&L)  is  the  Milestone  Decision  Authority  (MDA)  for  ACAT  ID  programs  and  ACAT  lAM 
programs  that  have  not  been  delegated.  The  USD(AT&L)  conducts  DAB  Reviews  for  ACAT  ID  and  lAM 
programs  at  major  milestone  decision  points,  at  the  Full-Rate  Production  Decision  Review,  at  Interim 
Program  Reviews,  and  at  other  times  as  necessary. 

1.3.3  DASD(DT&E) 

In  accordance  with  DoD  Instruction  (DoDI)  5134.17  (Reference  (a)),  the  DASD(DT&E)  shall  be  the  focal 
point  for  all  policy,  practice,  procedures,  and  acquisition  workforce  issues  relating  to  DT&E  within  DoD. 
The  DASD(DT&E)  is  the  principal  advisor  to  the  SecDef  and  the  USD(AT&L)  on  DT&E  in  the  DoD  in 
accordance  with  section  139b  of  Title  10,  United  States  Code  (U.S.C.)  (Reference  (b)).  See  Figure  1-1. 

1.3. 3.1  Responsibilities  of  the  DASD(DT&E) 

Some  of  the  key  duties  of  the  DASD(DT&E)  include: 

•  Develops  policies  and  guidance  for  the  following: 

o  The  planning,  execution,  and  reporting  of  DT&E  in  the  DoD,  including  integration  and 
DT&E  of  software; 

o  The  integration  of  developmental  and  operational  tests  in  coordination  with  the  DOT&E; 

o  The  planning,  execution,  and  reporting  of  DT&E  executed  jointly  by  more  than  one 
Military  Department  or  Defense  Agency; 

o  The  use  of  DT&E  planning  principles  and  best  practices;  and 

o  The  development  of  test  and  evaluation  strategies  (TESs)  and  test  and  evaluation  master 
plans  (TEMPs)  in  conjunction  with  the  DOT&E. 

•  Provides  guidance  to  defense  acquisition  programs  for  developing  and  documenting  the 
program's  evaluation  strategy  and  management  approach  in  the  TES  and  TEMP  throughout  the 
program's  life  cycle. 
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•  Reviews  and  approve  DT&E  portions  in  the  TES  and  TEMP  for  each  program  on  the  OSD  T&E 
oversight  list. 

•  Monitor  and  review,  beginning  with  the  materiel  development  decision,  the  DT&E  and 
development  planning  activities  of  the  pre-MDAPs  and  pre-MAIS  programs. 

•  Monitor  and  review  the  DT&E  activities  of  all  MDAPs  and  MAIS  programs  and  special  interest 
programs  identified  on  the  OSD  T&E  oversight  list. 

•  In  accordance  with  (I AW)  DTM  09-027  (Reference  (ac)),  ensure  access  to  all  test  data  and 
program  information  relevant  to  the  execution  of  testing  and  fulfillment  of  DASD(DT&E) 
responsibilities. 

•  Serves  as  the  DoD  functional  leader  for  the  T&E  acquisition  career  field.  Provide  advocacy, 
oversight,  and  guidance  to  the  acquisition  workforce  responsible  for  T&E. 

•  As  an  advisory  member  of  the  DAB  and  other  key  acquisition  bodies,  provides  independent 
assessments,  planned  and  ad  hoc,  of  programs'  DT&E,  execution,  and  risk. 

•  Periodically  review  the  organizations  and  capabilities  of  the  Military  Departments  with  respect 
to  DT&E.  Identify  needed  changes  or  improvements  to  such  organizations  and  capabilities,  and 
provide  input  regarding  needed  changes  or  improvements  for  the  T&E  strategic  plan. 

•  Submit  to  the  congressional  defense  committees  an  annual  report,  jointly  with  the  DASD(SE). 

•  Conduct  an  independent  assessment  of  operational  test  readiness  (AOTR)  for  all  MDAPs  and 
special  interest  programs. 

•  Jointly  with  the  DOT&E,  and  in  consultation  with  the  T&E  executives  of  the  cognizant  DoD 
Components,  determine  the  programs  designated  for  OSD  T&E  oversight. 

•  Serve  concurrently  as  the  Director,  Test  Resource  Management  Center  (TRMC),  and  in  this 
capacity,  report  directly  to  the  USD(AT&L)  (as  illustrated  in  Figure  1-1)  in  accordance  with 
section  196(f)  of  reference  (b). 

1.3. 3.2  DASD  (DT&E)  and  Service  Reports 

During  the  testing  of  ACAT  I  and  designated  weapon  systems,  the  DASD(DT&E)  and  Services  interaction 
includes  the  following  reporting  requirements: 

•  A  TES  must  be  provided  for  Milestone  (MS)  A  and  a  TEMP  (either  preliminary  or  updated,  as 
appropriate)  must  be  provided  for  consideration  and  approval  before  each  milestone  review, 
starting  with  MS  B. 

•  A  technical  assessment  of  DT&E  is  provided  to  the  ASD(R&E)  and  DOT&E  listing  the  T&E  results, 
conclusions,  and  recommendations  prior  to  a  milestone  decision  or  the  final  decision  to  proceed 
to  BLRIP. 
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1.3.4  DOT&E 

The  DOT&E  reports  to  the  SecDef  (as  illustrated  in  Figure  1-1)  and  has  special  reporting  requirements  to 
Congress.  The  DOT&E  must  provide  to  Congress  independent  analysis  and  insight  into  the  operational 
effectiveness,  suitability,  and  survivability  of  new  weapon  systems. 

1. 3.4.1  Duties  and  Authority  of  the  DOT&E 

Some  of  the  key  duties  and  authority  of  the  DOT&E  as  outlined  in  DoD  Directive  (DoDD)  5141.2 
(Reference  (c));  DODI  5000.02  (Reference  (d));  and  sections  139 , 2366 , 2399 ,  and  2400  of  Reference  (b) 
are  as  follows: 

•  Comply  with  requests  from  Congress  for  information  relating  to  operational  OT&E  in  the  DoD. 

•  In  conjunction  with  the  DASD(DT&E),  co-approve  the  TEMP  and  TES  for  major  and  other 
designated  defense  acquisition  programs,  special  interest  programs,  and  other  designated 
automated  information  system  (AIS)  programs. 

•  Approve  operational  test  plans,  and  OT&E  portions  of  test  planning  documents  for  major  and 
other  designated  defense  acquisition  programs,  special  interest  programs,  and  major  and  other 
designated  AIS  programs. 

•  Approve  the  TEMP  and  T&E  portions  of  integrated  program  management  documents  for 
programs  that  are  solely  under  DOT&E  oversight.  Approve  test  plans  for  operational  test  events 
of  acquisition  systems  under  DOT&E  oversight. 

•  Approve  LFT&E  strategies  and  management  plans  and,  if  developed  in  support  of  waivers  of  full- 
up  system-level  LFT,  alternative  LFT&E  strategies. 

•  Following  consultation  with  the  PM,  determine  the  number  of  production  or  production- 
representative  test  articles  required  for  LFT&E  and  lOT&E  of  programs  on  the  OSD  T&E  oversight 
list. 

•  Obtain  reports  and  information,  as  necessary  in  carrying  out  assigned  responsibilities  and 
functions. 

•  Provide  independent  oversight,  independent  evaluation,  and  objective  reporting  of  the  results 
of  operational  test  and  LFT&E. 

•  Determine  adequacy  of  operational  test  capabilities. 

•  Monitor  all  OT&E  activities. 

•  Coordinate  Military  Services  planning  and  execution  of  OT&E. 

•  Review  and  make  recommendations  to  the  SecDef  on  all  budgetary  and  financial  matters. 
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•  Provide  early  involvement  in  the  acquisition  cycle  (integrated  product  team  (IPT),  informal 
reviews,  etc.). 

•  Submit  special  reports  to  the  SecDef  and  Congress. 

1. 3.4.2  DOT&E  and  Service  Interactions 

For  DoD  and  DOT&E-designated  oversight  acquisition  programs,  the  Service  provides  the  DOT&E  the 
following: 

•  A  draft  copy  of  the  operational  test  plan  for  review  and  approval. 

•  Significant  test  plan  changes. 

•  The  final  Service  lOT&E  report,  which  must  be  submitted  to  DOT&E  before  the  Full-Rate 
Production  Decision  Review  (FRPDR)  for  incorporation  in  the  BLRIP  report. 

•  The  LFT&E  plan  for  approval,  and  the  Service  LFT  report  for  review. 

1.3.5  Defense  Information  Systems  Agency  (DISA) 

The  DISA  is  responsible  for  planning,  engineering,  acquiring,  testing,  fielding,  and  supporting  global  net- 
centric  information  and  communications  solutions  to  serve  the  needs  of  the  President,  the  Vice 
President,  the  SecDef,  and  the  DoD  Components,  under  all  conditions  of  peace  and  war.  The  DISA 
supports  national  security  communications  requirements  as  identified  in  the  National  Security 
Presidential  Directive  28  (Reference  (e))  and  Executive  Order  12472  (as  amended)  (Reference  (f)). 

In  accordance  with  DoDD  5105.19  (Reference  (g)),  the  DISA  maintains  a  major  field  independent 
operational  test  capability  (See  Figure  1-1)  through  the  Joint  Interoperability  Test  Command  (JITC). 
Under  the  direction  of  the  Director,  DISA,  the  JITC  conducts  operational  test  and  evaluation,  consistent 
with  DoD  Directive  5000.1  (Reference  (h))  and  Reference  (d) . 


1.4  Service  T&E  Management  Structures 
1.4.1  Army  T&E  Organizational  Relationship 

Army  Regulation  (AR)  73-1  (Reference  (i))  prescribes  implementing  policies  and  assigns  responsibilities 
for  T&E  activities  during  the  systems  acquisition  processes.  It  applies  to  all  systems  (materiel  and 
command,  control,  communications,  and  computers  (C4),  intelligence  (I),  and  information  technology 
(IT)  (C4I/IT)  developed,  evolved,  acquired,  and  managed  under  the 

auspices  of  AR  701  and  the  DAG  (Reference  (j)).  Reference  (i)  applies  to  Army  participation  in  joint  test 
and  evaluation  (JT&E)  and  multi-Service  operational  test  and  evaluation  (MOT&E).  The  Army 
management  structure  for  T&E  is  depicted  in  Figure  1-2. 
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Figure  1-2  Army  T&E  Organization 

1.4. 1.1  Army  Acquisition  Executive  (AAE) 

The  Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technology  (ASA(ALT))  has  the 
principal  responsibility  for  all  Department  of  the  Army  matters  and  policy  related  to  acquisition, 
logistics,  technology,  procurement,  the  industrial  base,  and  security  cooperation.  Additionally,  the 
ASA(ALT)  serves  as  the  AAE.  The  AAE  administers  acquisition  programs  by  developing/promulgating 
acquisition  policies  and  procedures  as  well  as  appointing,  supervising,  and  evaluating  assigned  program 
executive  officers  (PEOs)  and  direct-reporting  PMs  . 

1.4. 1.2  Army  T&E  Executive 

The  Army  T&E  Executive  establishes,  reviews,  enforces,  and  supervises  Army  T&E  policy  and  procedures 
including  overseeing  all  Army  T&E  associated  with  the  system  research,  development,  and  acquisition  of 
all  materiel  systems  and  C4/IT  systems.  As  delegated  by  the  AAE,  the  Army  T&E  Executive  is  the  sole 
Headquarters,  Department  of  the  Army  (HQDA)  approval  authority  for  TEMPs. 

The  Test  and  Evaluation  Office  within  the  Office  of  the  Deputy  Under  Secretary  of  the  Army,  known  as 
the  Deputy  Under  Secretary  of  the  Army  for  Test  and  Evaluation  (DUSA-TE),  provides  support  for  the 
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Army  T&E  Executive.  In  this  capacity,  it  has  the  mission  to  establish  policy  and  resources  that  are 
disciplined  and  flexible  enough  to  support  safe  and  reliable  equipment  for  the  current  and  future  Army 
and  DoD  chemical  and  biological  defense.  DUSA-TE  also  provides  T&E  subject  matter  expertise  and 
oversight  of  Army  and  DoD  chemical  and  biological  acquisition  programs  and  represents  Army  T&E 
interests  at  OSD  and  tri-Service  committees  and  forums. 

1.4.1.3  Army  Test  and  Evaluation  Command  (ATEC) 

ATEC  supports  the  systems  acquisition,  force  development,  and  experimentation  processes  through 
overall  management  of  the  Army's  T&E  programs.  In  this  role,  ATEC  manages  the  Army's  developmental 
and  operational  testing,  all  system  assessments  and  evaluations,  and  management  of  joint  T&E.  ATEC  is 
the  Army's  independent  operational  test  agency  (OTA)  reporting  directly  to  the  Vice  Chief  of  Staff  of  the 
Army  (VCSA)  through  the  Director  of  the  Army  Staff  (DAS). 

ATEC  has  the  primary  responsibility  for  conducting  (DTs)  for  the  Army.  Responsibilities  include  the 
following: 

•  Perform  the  duties  of  Government  developmental  tester  for  all  Army  systems  except  for  those 
systems  assigned  to  the  Communications-Electronics  Research,  Development,  and  Engineering 
Center  of  the  Army  Materiel  Command  (AMC)  Research,  Development  and  Engineering 
Command  (RDECOM)  by  HQDA  (Deputy  Chief  of  Staff,  G-6);  Medical  Command  (MEDCOM); 
Intelligence  and  Security  Command  (INSCOM);  Space  and  Missile  Defense  Command  (SMDC); 
and  Army  Corps  of  Engineers  (ACE). 

•  Provide  test  facilities  and  testing  expertise  in  support  of  the  acquisition  of  Army  and  other 
defense  systems. 

•  Operate  and  maintain  the  Army's  portion  of  the  Major  Range  and  Test  Facility  Base  (MRTFB) 
(except  for  the  United  States  Army  Kwajalein  Atoll  in  accordance  with  DoDD  3200.11 , 
(Reference  (k)). 

•  Provide  testers  with  a  safety  release  for  systems  before  the  start  of  pretest  training  for  tests 
that  use  Soldiers  as  test  participants. 

•  Provide  safety  confirmations  for  milestone  acquisition  decision  reviews  and  the  materiel  release 
decision. 

•  Manage  the  Army  Test  Incident  Reporting  System. 

1.4. 1.3.1  Army  Operational  Testers 

The  U.S.  Army  Operational  Test  Command  (OTC)  within  ATEC  has  the  primary  responsibility  for 
conducting  most  Operational  Tests  (OTs)  for  the  Army  and  supporting  Army  participation  in  Joint  T&E. 
OTC  responsibilities  include  the  following: 
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•  Perform  the  duties  of  operational  tester  for  all  Army  systems  except  for  those  systems  assigned 
to  MEDCOM,  INSCOM,  SMDC,  and  United  States  Army  Special  Operations  Command  (USASOC). 

•  Perform  the  duties  of  operational  tester  for  assigned  multi-Service  OTs  and  (on  a  customer 
service  basis)  for  Training  and  Doctrine  Command  (TRADOC)  force  development  tests  and/or 
experimentation  (FDT/E). 

•  Provide  test  facilities  and  testing  expertise  in  support  of  the  acquisition  of  Army  and  other 
defense  systems,  and  for  other  customers  on  a  cost-reimbursable  and  as-available  basis. 

•  Program  and  budget  the  funds  to  support  OT  except  out-of-cycle  tests  (which  are  usually  paid 
for  by  the  proponent). 

•  Develop  and  submit  OT  and  FDT/E  test  resource  plans  (TRPs)  to  the  Army's  Test  Schedule  and 
Review  Committee  (TSARC). 

1.4. 1.3. 2  Army  Evaluation  Center  (AEC) 

The  AEC  is  an  independent  subordinate  element  within  ATEC  that  has  the  primary  responsibility  for 
conducting  Army  system  evaluations  and  system  assessments  in  support  of  the  systems  acquisition 
process.  Decision  makers  use  AECs  independent  report  addressing  an  Army  systems  operational 
effectiveness,  suitability,  and  survivability.  AEC  responsibilities  include  the  following: 

•  Perform  the  duties  of  system  evaluator  for  all  Army  systems  except  for  those  systems  assigned 
to  MEDCOM,  INSCOM,  and  the  commercial  items  (Cl)  assigned  to  ACE;  conduct  continuous 
evaluation  (CE)  on  all  assigned  systems. 

•  Develop  and  promulgate  evaluation  capabilities  and  methodologies. 

•  Coordinate  system  evaluation  resources  through  the  TSARC. 

•  Preview  programmed  system  evaluation  requirements  for  possible  use  of  modeling  and 
simulation  (M&S)  to  enhance  evaluation  and  reduce  costs. 

•  Perform  manpower  and  personnel  integration  assessments  in  coordination  with  Deputy  Chief  of 
Staff,  G-1  Army  Research  Laboratory  (ARL)-Fluman  Research  and  Engineering  Directorate. 

•  Perform  the  integrated  logistics  support  (ILS)  program  surveillance  for  Army  systems.  Perform 
independent  logistics  supportability  assessments  and  report  them  to  the  Army  logistician  and 
other  interested  members  of  the  acquisition  community  . 
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1.4.2  NavyT&E  Organizational  Relationship 


Figure  1-3  Navy  Test  and  Evaluation  Organization 

The  organizational  structure  forT&E  in  the  Navy  is  illustrated  in  Figure  1-3.  Within  the  Navy  Secretariat, 
the  Secretary  of  the  Navy  (SECNAV)  has  assigned  general  and  specific  research,  development,  test,  and 
evaluation  (RDT&E)  responsibilities  to  the  Assistant  Secretary  of  the  Navy  for  Research,  Development, 
and  Acquisition  (ASN(RD&A))  and  to  the  Chief  of  Naval  Operations  (CNO).  The  CNO  has  responsibility  for 
the  TEMP  process.  T&E  policy  and  guidance  are  exercised  through  the  Director,  Innovation,  Test  and 
Evaluation,  and  Technology  Requirements  (CNO  (N84)). 

1.4. 2.1  Navy  DT&E  Organizations 

The  Navy's  senior  systems  development  authority  is  divided  among  the  commanders  of  the  system 
commands  with  Naval  Air  Systems  Command  (NAVAIR)  developing  and  performing  DT&E  on  aircraft  and 
their  essential  weapon  systems;  Naval  Sea  Systems  Command  (NAVSEA)  developing  and  performing 
DT&E  on  ships,  submarines,  and  their  associated  weapon  systems;  and  Space  and  Naval  Warfare 
Systems  Command  (SPAWAR)  developing  and  performing  DT&E  on  all  other  systems.  System  acquisition 
is  controlled  by  a  chartered  PM  or  by  the  commander  of  a  systems  command.  In  both  cases,  the 
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designated  developing  agency  is  responsible  for  DT&E  and  for  the  coordination  of  all  T&E  planning  in  the 
TEMP.  Developing  agencies  are  responsible  for  the  following: 

•  Develop  test  issues  based  on  the  thresholds  established  by  user  requirements. 

•  Identify  the  testing  facilities  and  resources  required  to  conduct  the  DT&E. 

•  Develop  the  DT&E  test  reports. 

Reporting  directly  to  the  CNO,  the  President  of  the  Board  of  Inspection  and  Survey  is  responsible  for 
conducting  acceptance  trials  of  new  ships  and  aircraft  acquisitions  and  is  the  primary  Navy  authority  for 
Production  Acceptance  Test  and  Evaluation  (PAT&E)  of  these  systems. 

1.4. 2.2  Navy  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 

The  Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR)  commands  the  Navy's 
independent  OT&E  activity  and  reports  directly  to  the  CNO.  The  functions  of  the  OPTEVFOR  include  the 
following: 

•  Establish  early  liaison  with  the  developing  agency  to  ensure  an  understanding  of  the 
requirements  and  plans. 

•  Review  acquisition  program  documentation  to  ensure  that  documents  are  adequate  to  support 
a  meaningful  T&E  program. 

•  Plan  and  conduct  realistic  OT&E. 

•  Develop  tactics  and  procedures  for  the  employment  of  systems  that  undergo  OT&E  (as  directed 
by  the  CNO). 

•  Provide  recommendations  to  the  CNO  for  the  development  of  new  capabilities  or  the  upgrade 
of  ranges. 

•  Conduct  OT&E  on  Marine  Corps  aviation  systems. 


1.4.3  Marine  Corps  T&E  Organizational  Relationship 

The  Office  of  Program,  Budget,  and  Execution,  Fleadquarters  Marine  Corps,  directs  the  total  Marine 
Corps  program  effort  to  support  the  acquisition  of  new  systems.  The  Marine  Corps  does  not  have  a 
position  that  is  analogous  to  that  of  the  Director,  Innovation,  T&E  and  Technology  Requirements  (CNO 
(N84)).  The  Commanding  General  (CG),  Marine  Corps  Systems  Command  (MCSC)  is  the  sponsor  for 
RDT&E  and  is  responsible  for  DT&E  within  the  Marine  Corps.  The  CG  MCSC  also  reports  directly  to  the 
ASN(RD&A).  Figure  1-3  illustrates  the  Marine  Corps  organization  for  T&E  management. 
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1.4. 3.1  Marine  Corps  DT&E  Organizations 

The  CG  MCSC  is  the  Marine  Corps  materiel  developing  agent  and  directly  interfaces  with  the  Navy 
Systems  Commands.  The  CG  MCSC  implements  policies,  procedures,  and  requirements  for  DT&E  of  all 
systems  acquired  by  the  Marine  Corps.  The  Marine  Corps  also  uses  DT&E  and  OT&E  performed  by  other 
Services,  which  may  develop  systems  of  interest  to  the  Corps. 

1.4. 3.2  Marine  Corps  Test  and  Evaluation  Activity  (MCOTEA) 

MCOTEA  is  responsible  for  the  independent  OT&E  of  assigned  Navy,  Marine  Corps,  and  joint  acquisition 
programs  to  ensure  that  all  events  are  effectively  planned,  conducted,  and  reported.  MCOTEA 
coordinates  the  scheduling  of  resources  for  OT  requiring  Marine  operating  forces  support  through  the 
Marine  forces  synchronization  conferences.  Aviation  programs  sponsored  by  the  CNO  undergo 
independent  OT&E  by  the  COMOPTEVFOR. 

1.4.4  Air  Force  T&E  Organizational  Relationships 

Air  Force  Policy  Directive  (AFPD)  99-1  (Reference  (I))  and  the  Air  Force  99-serices  of  Instructions 
establishes  policies  for  the  T&E  process  and  infrastructure. 

The  Assistant  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ)  is  the  senior-level  authority  for 
research,  development,  and  acquisition  (RDA)  within  the  Air  Force.  As  illustrated  in  Figure  1-4,  the 
SAF/AQ  is  an  advisor  to  the  Secretary  of  the  Air  Force  and  interfaces  directly  with  the  DASD(DT&E)  and 
the  DOT&E.  The  SAF/AQ  receives  DT&E  and  OT&E  results  as  a  part  of  the  acquisition  decision  process. 
Within  the  SAF/AQ  structure,  there  is  a  military  deputy  (acquisition)  who  is  the  Air  Force  primary  staff 
officer  with  responsibility  for  RDA.  The  Director,  Air  Force  Test  and  Evaluation  (AF/TE),  provides  T&E 
policy  and  oversight  and  reports  directly  to  the  Chief  of  Staff  of  the  Air  Force  (CSAF).  AF/TE  processes 
DT&E  and  OT&E  documentation,  resolves  T&E  issues  for  the  Air  Force,  and  manages  the  review  of  the 
TEMP. 

1.4.4.1  Air  Force  DT&E  Organization 

The  Air  Force  Materiel  Command  (AFMC)  and  Air  Force  Space  Command  (AFSPC)  are  implementing 
commands  that  conduct  Government  DT&E  and  manage  acquisition  programs.  AFMC  and  AFSPC 
perform  all  levels  of  research;  develop  weapons  systems,  support  systems,  and  support  equipment. 
Within  these  implementing  commands  are  product  centers,  logistics  centers,  test  centers,  and 
laboratories  as  well  as  a  wide  variety  of  test  ranges. 

Qnce  the  weapon  system  is  fielded,  AFMC  and  AFSPC  retain  management  responsibility  for  developing 
and  testing  system  improvements  and  upgrades.  The  AFSC  is  responsible  for  DT&E  of  space  and  missile 
systems . 
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Figure  1-4  Air  Force  Test  and  Evaluation  Organization 
I.4.4.2.  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC) 

As  the  Air  Force  OTA,  AFOTEC  conducts  independent  OT&E  of  all  programs  on  the  OSD  OT&E  Oversight 
List  and  other  programs  as  directed.  AFOTEC  also  conducts  OTs  in  support  of  joint  urgent  operational 
needs  and  Warfighter  rapid  acquisition  programs.  The  AFOTEC  commander  reports  directly  to  the  CSAF. 
In  preparation  for  OT&E,  AFOTEC  reviews  all  relevant  operational  and  training  requirements, 
employment  and  maintenance  concepts,  and  tactics.  Operational  and  implementing  major  commands 
(MAJCOMs)  provide  operational  concepts,  personnel,  and  resources  to  assist  AFOTEC  in  performing 
OT&E. 

1.4.4.3  MAJCOM  Operational  Test  Organizations 

OT  organizations  (squadrons  and  wings)  are  embedded  in  each  operating  MAJCOM  to  conduct  follow-on 
operational  test  and  evaluation  (FOT&E)  for  systems  past  AFOTEC-conducted  lOT&E  and  to  conduct 
OT&E  on  all  systems  in  sustainment.  Force  development  evaluations  (FDEs),  which  are  a  type  of  OT&E, 
are  performed  by  MAJCOM  OT  organizations  in  support  of  MAJCOM-managed  system  acquisition- 
related  decisions  prior  to  initial  fielding  or  for  MAJCOM  sustainment  or  upgrade  activities. 
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1.5  Summary 

An  increased  emphasis  on  T&E  has  placed  greater  demands  on  the  OSD  and  DoD  Components  to 
carefully  structure  organizations  and  resources  to  ensure  maximum  effectiveness.  Renewed  interest  by 
Congress  in  testing  as  a  way  of  assessing  systems  utility  and  effectiveness,  reports  by  a  variety  of 
organizations  on  improving  acquisition  effectiveness,  and  continuing  acquisition  reform  initiatives  have 
resulted  in  reorganizations  within  the  DoD  and  the  Services  to  improve  the  management  of  T&E 
resources,  planning,  execution,  and  reporting,  in  support  of  acquisition  programs. 
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Test  &  Evaluation  Management  Guide 
Chapter  2  -  The  T&E  Process 

2.1  Introduction 

The  fundamental  purpose  of  T&E  is  to  provide  essential  information  to  decision  makers,  verify  and 
validate  performance  capabilities  documented  as  requirements,  assess  attainment  of  technical 
performance  parameters,  and  determine  whether  systems  are  operationally  effective,  suitable, 
survivable,  and  safe  for  intended  use.  During  the  early  phases  of  development,  T&E  is  conducted  to 
demonstrate  the  feasibility  of  conceptual  approaches,  evaluate  design  risk,  identify  design  alternatives, 
compare  and  analyze  trade-offs,  and  estimate  satisfaction  of  operational  requirements.  As  a  system 
undergoes  design  and  development,  the  iterative  process  of  testing  moves  gradually  from  a 
concentration  on  DT&E,  which  is  concerned  chiefly  with  attainment  of  engineering  design  goals  and 
verification  of  technical  specifications,  to  increasingly  comprehensive  OT&E,  which  focuses  on  questions 
of  operational  effectiveness,  suitability,  and  survivability.  Although  there  are  usually  separate  DT  and  OT 
events,  DT&E  and  OT&E  are  not  necessarily  serial  phases  in  the  evolution  of  a  weapon  system 
development.  Integrated  DT  and  OT  are  encouraged  when  appropriate;  i.e.,  conferring  possible  cost  or 
time  savings. 

The  discipline  of  T&E  has  its  origins  in  the  testing  of  hardware.  This  tradition  is  heavily  embedded  in  its 
vocabulary  and  procedures.  The  advent  of  software-intensive  systems  has  brought  new  challenges  to 
testing,  and  new  approaches  are  discussed  in  Chapter  15.  Remaining  constant  throughout  the  T&E 
process,  whether  testing  hardware  or  software,  is  the  need  for  thorough,  logical,  systematic,  and  early 
test  planning  followed  by  feedback  of  well-documented  and  unbiased  T&E  results  to  system  developers, 
users,  and  decision  makers. 

T&E  provides  useful  information  to  many  customers  and  value  to  the  PM.  DT&E  and  OT&E  must  be 
integrated  into  an  efficient  continuum  of  testing  as  much  as  practical  without  compromising  the 
objectives  of  either  T&E  activity.  Information  assurance  (lA)  and  interoperability  testing  as  well  as 
reliability  analysis,  planning,  tracking,  and  reporting  must  also  be  integrated  into  this  continuum. 


2.2  Defense  Systems  Acquisition  Process 

The  primary  objective  of  the  defense  acquisition  process  is  the  acquisition  of  quality  products  that 
satisfy  user  needs  with  measurable  improvements  to  mission  capability  and  operational  support,  in  a 
timely  manner,  and  at  a  fair  and  reasonable  cost.  As  it  is  now  structured,  the  defense  system  life  cycle  is 
to  replicate  the  preferred  acquisition  strategy  of  an  evolutionary  acquisition  process  that  uses 
incremental  development  or  rapid  acquisition  processes.  The  three  major  elements-pre-system 
acquisition,  system  acquisition,  and  sustainment-may  include  the  following  five  phases: 

1.  Materiel  Solution  Analysis  (MSA). 

2.  Technology  Development  (TD). 

3.  Engineering  and  Manufacturing  Development  (EMD). 


23 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcms 


/KI-L0J!lb 


4.  Production  and  Deployment  (P&D). 

5.  Operations  and  Support  (O&S). 

As  Figure  2-1  shows,  these  phases  are  separated  by  key  decision  points  when  a  MDA  reviews  a  program, 
considers  its  development  maturity,  and  determines  authorization  to  advance  to  the  next  phase  in  the 
cycle.  Thus,  T&E  planning  and  test  results  play  an  important  part  in  the  MDA  review  process. 


DEFENSE  ACQUISITION  MODEL 
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Figure  2-1  Defense  Acquisition  System  Model 

USD(AT&L)  memorandum  (Reference  (m))  provided  supplementary  guidance  which  introduced 
procedural  changes  to  the  acquisition  model.  These  changes  included,  for  example,  the  addition  of  a 
Pre-EMD  phase  as  well  as  directing  that  the  substance  of  milestone  reviews  (e.g.  SEPs  and  TEMPs)  be 
addressed  earlier  at  each  point  of  the  milestone  decision  process.  Figure  2-2  shows  this  process  and  its 
relationship  with  T&E  processes.  USD(AT&L)  memorandum  (Reference  (m))  provided  supplementary 
guidance  which  introduced  procedural  changes  to  the  acquisition  model.  These  changes  included,  for 
example,  the  addition  of  a  Pre-EMD  phase  as  well  as  directing  that  the  substance  of  milestone  reviews 
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(e.g.  SEPs  and  TEMPs)  be  addressed  earlier  at  each  point  of  the  milestone  decision  process.  Figure  2-2 
shows  this  process  and  its  relationship  with  T&E  processes. 
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Figure  2-2  Improving  Milestone  Process  Effectiveness 


2.2.1  The  T&E  Contribution  at  Major  Milestones 

The  following  description  of  the  defense  system  acquisition  process,  summarized  from  Reference  (d), 
shows  how  T&E  fits  within  the  context  of  the  larger  process. 

T&E  progress  is  monitored  by  OSD  throughout  the  acquisition  process.  OSD  oversight  extends  to  MDAPs, 
MAIS  programs,  or  designated  acquisition  programs  to  include  defense  business  systems  as  defined  in 
Directive-Type  Memorandum  (DTM)  11-009  (Reference  (n)).  T&E  officials  within  OSD  render 
independent  assessments-based  on  participation  in  test  events,  direct  observation  of  test  events,  and 
continuous  oversight  of  activities-to  the  DAB,  the  DAE,  and  the  SecDef  at  each  system  milestone  review. 
These  assessments  are  based  on  the  following  T&E  information: 
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Test  planning  documents,  such  as  the  TEMP  and  more  detailed  supporting  documents,  developed  by  the 
responsible  DoD  Component  activities. 

Test  reports  and  briefings  prepared  by  the  Component  test  activities. 

Data  from  tests,  M&S,  and  data  from  other  sources  such  as  Component  PMs,  laboratories,  industry 
developers,  studies,  and  analyses. 

At  MS  B,  the  OSD  T&E  assessments  reflect  an  evaluation  of  system  concepts  and  technology  alternatives 
using  early  performance  parameter  objectives  and  thresholds  found  in  an  approved  preliminary  TEMP. 
At  MS  C,  assessments  include  an  evaluation  of  system-level  test  plans  and  test  results.  At  the  FRPDR, 
assessments  include  consideration  of  the  operational  effectiveness  and  suitability  evaluations  of  low- 
rate  initial  production  (LRIP)  systems. 

In  accordance  with  DTM  11-003  (Reference  (o)),  PMs  formulate  a  comprehensive  reliability,  availability, 
and  maintainability  (RAM)  program  using  an  appropriate  reliability  growth  strategy  to  improve  RAM 
performance  until  RAM  requirements  are  satisfied.  The  program  consists  of  engineering  activities 
including  RAM  allocations,  block  diagrams,  and  predictions;  failure  definitions  and  scoring  criteria; 
failure  mode,  effects,  and  criticality  analysis  (FMECA);  maintainability  and  built-in  test  demonstrations; 
reliability  growth  testing  at  the  system  and  subsystem  level;  and  a  failure  reporting  and  corrective  action 
system  maintained  through  design,  development,  production,  and  sustainment.  In  accordance  with 
USD(AT&L)  Memorandum  (Reference  (p)),  it  is  DoD  policy  for  programs  to  be  formulated  to  execute  a 
viable  RAM  strategy  that  includes  a  reliability  growth  program  as  an  integral  part  of  design  and 
development.  The  RAM  program  is  an  integral  part  of  the  systems  engineering  (SE)  process. 

A  key  contribution  made  by  T&E  in  the  acquisition  process  is  the  early  detection  and  reporting  of 
deficiencies  that  may  adversely  impact  the  performance  capability  or  availability  and  supportability  of  a 
system.  A  comprehensive  and  repeatable  deficiency  reporting  process  should  be  used  throughout  the 
acquisition  process  to  report,  evaluate,  and  track  system  deficiencies  and  to  provide  the  impetus  for 
corrective  actions  that  improve  performance  to  desired  levels.  The  lead  DoD  Component  and  the  PM,  or 
equivalent,  prepares  a  preliminary  RAM  and  cost  rationale  report  in  support  of  the  MS  A  decision. 

In  accordance  with  Reference  (d),  there  shall  be  only  one  MS  B  per  program  or  evolutionary  increment. 
Each  increment  of  an  evolutionary  acquisition  shall  have  its  own  MS  B  unless  the  MDA  determines  that 
the  increment  will  be  initiated  at  MS  C. 

2.2.2  MSA  and  TD  Phases 

A  defense  pre-system  acquisition  process  begins  with  the  requirements  process  identification  of  a 
materiel  need  and  the  development  of  the  Initial  Capabilities  Document  (ICD).  A  decision  is  made  to 
start  MSA  phase,  and  the  Technology  Development  Strategy  (TDS)  evolves  with  the  development  of 
initial  test  planning  concepts  in  the  TES.  MSA  ends  when  the  MDA  approves  the  preferred  solution 
resulting  from  the  Analysis  of  Alternatives  (AoA),  identifying  measures  of  effectiveness  (MOEs),  and 
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approves  the  associated  TDS.  NOTE:  A  useful  reference  on  such  items  as  MOEs  and  measures  of 
performance  (MOPs)  and  their  relationship  to  the  overall  technical  measurement  process  including 
technical  performance  measurement  (TPM)  can  be  found  in  International  Council  on  Systems 
Engineering  (INCOSE)-TP-2003-020-01  (Reference  (q))  which  is  a  technical  report  developed  as  a 
collaborative  effort  between  the  DoD  Practical  Software  and  Systems  Measurement  organization  and 
INCOSE.  The  TD  phase  follows  a  MS  A  decision  during  which  the  risks  of  the  relevant  technologies  of  the 
alternative  selected  for  satisfying  the  user's  needs  are  investigated.  Shortly  after  the  milestone  decision, 
an  integrated  team  begins  transitioning  the  test  planning  in  the  TES  into  the  evaluation  strategy  for 
formulation  of  a  TEMP. 

The  integrated  team  that  supports  the  PM  and  program  management  office  (PMO)  is  the  T&E  Working- 
level  Integrated  Product  Team  (WIPT).  The  T&E  WIPT  is  a  defined  forum  serving  the  PM  and  PMO  as  the 
T&E  subject  matter  experts  (SMEs)  responsible  for  supporting  the  PM  and  other  program  WIPTs  on  all 
aspects  of  a  programs  T&E  effort.  This  effort  includes  T&E  program  strategy,  design,  development, 
oversight,  analysis,  assessment,  and  reporting  of  test  results.  The  T&E  WIPT  should  be  established  and 
chartered  as  early  as  possible  around  MS  A  so  it  can  be  involved  in  program  strategy  discussions  and 
plans. 

The  lead  DoD  Component  and  the  PM,  or  equivalent,  should  prepare  a  preliminary  RAM  and  cost 
rationale  report  in  support  of  the  MS  A  decision.  This  report  provides  a  quantitative  basis  for  reliability 
requirements  and  improves  cost  estimates  and  program  planning.  The  report  should  be  attached  to  the 
Systems  Engineering  Plan  (SEP)  at  MS  A  and  updated  in  support  of  MS  B  and  MS  C. 

As  stated  in  the  ASD(R&E)  Guidance  (Reference  (r)),  a  Technology  Readiness  Assessment  (TRA)  is  a 
systematic,  metrics-based  process  that  assesses  the  maturity  of  and  the  risk  associated  with  critical 
technologies  to  be  used  in  MDAPs.  It  is  conducted  by  the  PM  with  the  assistance  of  an  independent 
team  of  SMEs.  It  is  provided  to  the  ASD(R&E)  and  will  provide  part  of  the  basis  upon  which  the  ASD(R&E) 
advises  the  MDA  at  MS  B  or  at  other  events  designated  by  the  MDA  to  assist  in  the  determination  of 
whether  the  technologies  of  the  program  have  acceptable  levels  of  risk-based  in  part  on  the  degree  to 
which  they  have  been  demonstrated  (including  demonstration  in  a  relevant  environment)-and  to 
support  risk-mitigation  plans  prepared  by  the  PM.  The  plan  for  conducting  a  TRA  is  provided  to  the 
ASD(R&E)  by  the  PM  upon  approval  by  the  Component  Acquisition  Executive  (CAE). 

A  TRA  is  required  by  Reference  (d)  for  MDAPs  at  MS  B  (or  at  a  subsequent  milestone  if  there  is  no  MS  B). 
It  is  also  conducted  whenever  otherwise  required  by  the  MDA.  A  TRA  is  required  for  space  systems  by 
DTM  09-25  (Reference  (s)).  The  TRA  final  report  for  MDAPs  must  be  submitted  to  the  ASD(R&E)  for 
review  to  support  the  requirement  that  the  ASD(R&E)  provide  an  independent  assessment  to  the  MDA. 

A  TRA  focuses  on  the  programs  critical  technologies  (i.e.,  those  that  may  pose  major  technological  risk 
during  development,  particularly  during  the  EMD  phase  of  acquisition).  Technology  Readiness  Levels 
(TRLs)  can  serve  as  a  helpful  knowledge-based  standard  and  shorthand  for  evaluating  technology 
maturity,  but  they  must  be  supplemented  with  expert  professional  judgment. 
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To  reduce  the  risk  associated  with  entering  EMD,  Reference  (d)  requires  requests  for  proposals  (RFPs)  to 
incorporate  language  that  prevents  the  award  of  an  EM  D  contract  if  it  includes  technologies  that  have 
not  been  demonstrated  adequately.  Adequate  demonstration  in  a  relevant  environment  (TRL  6)  is  one 
benchmark  that  is  evaluated,  but  it  is  not  the  only  consideration,  nor  necessarily  dispositive.  As  such,  a 
generic  TRA  not  based  on  the  planned  specific  technical  solution  is  insufficient.  Since  the  TRA  must  be 
based  on  the  technologies  of  the  program  that  entail  some  element  of  risk,  TRAs  may  have  to  be 
performed  on  all  the  competitors'  proposals  in  a  source  selection. 

The  TRAs  described  in  USD(AT&L)  Memorandum  (Reference  (t))  replace  the  former  TRAs  described  in 
the  Director,  Research  Directorate,  Deskbook  (Reference  (u)).  TRAs  that  must  be  submitted  to  the 
ASD(R&E)  are  required  only  for  MDAPs  that  require  certification  under  section  2366b  of  Reference  (b)  or 
other  provisions  of  law,  or  when  otherwise  directed  by  the  MDA.  Generally,  TRAs  are  not  required  for 
MDAPs  at  MS  C.  Independent  of  the  elimination  of  the  formal  requirement  to  conduct  a  TRA  for  a  MAIS, 
MS  C,  and  ACAT  IIIV  programs,  all  PMs  and  their  chains  of  command  retain  complete  responsibility  for 
assessing,  managing,  and  mitigating  acquisition  program  technology  risk.  MDAs  for  non-ACAT  I  programs 
should  consider  requiring  TRAs  for  those  programs  when  technological  risk  is  present. 

The  TD  effort  concludes  with  a  decision  review  at  MS  B  when  an  affordable  increment  of  militarily  useful 
capability  has  been  identified,  the  technology  for  that  increment  has  been  demonstrated  in  a  relevant 
environment,  and  a  system  can  be  developed  for  production  within  a  reasonable  time  frame  or  the  MDA 
decides  to  terminate  the  effort. 

Typical  T&E-related  documents  at  the  MS  B  review  are  the  Acquisition  Decision  Memorandum  (ADM) 
(exit  criteria),  ICD,  Capability  Development  Document  (CDD)  (performance  parameters).  Acquisition 
Strategy,  System  Threat  Assessment  (STA),  an  Early  Operational  Assessment  (EOA),  and  the  TEMP. 
Additional  program  management  documents  prepared  before  MS  B  include  the  AoA,  Independent  Cost 
Estimate ,  and  the  concept  baseline  version  of  the  Acquisition  Program  Baseline  (APB),  which 
summarizes  the  weapons  functional  specifications,  performance  parameters,  and  cost  and  schedule 
objectives. 

The  program  office  for  major  programs  (OSD  oversight)  must  give  consideration,  if  relevant,  to 
requesting  a  waiver  for  full-up,  system-level  (FUSE)  LFT  and  identification  of  LRIP  quantities  for  lOT&E. 

2.2.3  T&E  Contributions  Prior  to  MS  B 

During  MSA  and  TD  activities  prior  to  MS  B,  laboratory  testing  and  M&S  are  conducted  by  the 
contractors  and  the  development  agency  to  demonstrate  and  assess  the  capabilities  of  key  subsystems 
and  components.  The  test  and  simulation  designs  are  based  on  the  operational  needs  documented  in 
the  ICD.  Studies,  analyses,  simulation,  and  test  data  are  used  by  the  development  agency  to  explore  and 
evaluate  alternative  concepts  proposed  to  satisfy  the  user's  needs.  Also  during  this  period,  the  OTA 
monitors  MSA  and  TD  activities  to  gather  information  for  future  T&E  planning  and  to  provide 
effectiveness  and  suitability  input  desired  by  the  PM.  The  OTA  also  conducts  EOAs,  as  feasible,  to  assess 
the  operational  impact  of  candidate  technical  approaches  and  to  assist  in  selecting  preferred  alternative 
system  concepts. 
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The  development  agency  prepares  the  TDS  in  conjunction  with  the  early  T&E  strategy  for  DT&E  and 
M&S.  The  TES  presents  T&E  plans  for  system  design(s)  engineering  and  performance  evaluations.  The 
strategies  also  include  the  tasks  and  processes  to  be  stated  in  the  REP  that  the  contractor  will  be 
required  to  employ  to  demonstrate  the  achievement  of  reliability  design  requirements.  The  TES  and 
TEMP  specify  how  reliability  will  be  tested  and  evaluated  during  the  associated  acquisition  phase.  The 
OTA  may  provide  an  EOA.  This  information  is  incorporated  into  the  PMs  TEMP  that  documents  the 
programs  T&E  strategy  that  supports  the  MS  B  decision  to  proceed  to  the  next  phase. 

2.2.4  EMD  Phase 

The  MS  B  decision  is  program  initiation  for  systems  acquisition;  establishes  broad  objectives  for  program 
cost,  schedule,  and  technical  performance;  and  starts  the  EMD  phase  of  the  acquisition  life  cycle.  The 
EMD  phase  is  divided  into  two  sub  phases:  Integrated  Systems  Design  (ISD)  and  System  Capability  and 
Manufacturing  Process  Demonstration  (SC&MPD). 

After  the  MS  B  decision  for  a  program  start,  the  ISD  work  effort  begins  during  which  a  selected  concept, 
typically  brassboards  or  early  prototypes,  is  refined  through  SE,  analysis,  and  design.  SE  must  manage  all 
requirements  as  an  integrated  set  of  design  constraints  that  are  allocated  down  through  the  various 
levels  of  design  and  other  technical  activities  (see  example  in  Figure  2-2).  This  work  effort  ends  when  the 
integration  of  the  system  components  during  DT  of  prototypes  demonstrates  adequate  maturity  and 
readiness  to  either  enter  into  SC&MPD  or  make  a  change  to  the  acquisition  strategy. 

The  SC&MPD  work  effort  is  intended  to  demonstrate  the  ability  of  the  system  to  operate  consistent  with 
the  approved  key  performance  parameters  (KPPs).  Work  advances  the  design  to  an  engineering 
development  model  (EDM)  that  is  evaluated  for  readiness  to  enter  LRIP.  The  EDM  should  have 
demonstrated  acceptable  performance  in  DT&E  and  the  Operational  Assessment  (OA),  with  acceptable 
interoperability  and  operational  supportability.  DT&E  results  include  systems  meeting  KPPs  and  key 
system  attributes  (KSAs)  as  part  of  the  exit  criteria  for  EMD. 

Preparation  for  the  MS  C  decision  establishes  more  refined  cost,  schedule,  and  performance  objectives 
and  thresholds,  and  the  Capability  Production  Document  (CPD)  establishes  the  criteria  for  testing  of  LRIP 
items.  Documents  of  interest  to  the  T&E  manager  at  the  time  of  the  MS  C  review  include  the  ADM  (exit 
criteria),  updated  TEMP,  updated  STA,  AoA,  Development  Baseline,  DT  results,  and  the  OA. 
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Figure  2-3  Key  Technical  Activities  and  Technical  Reviews/Audits  during  the  Acquisition  Lifecycle 
2.2.4.1  T&E  Contributions  During  ISD 

During  the  EMD  phase,  concepts  approved  for  prototyping  form  the  baseline  that  is  used  for  detailed 
test  planning.  The  design  is  matured  into  an  EDM,  which  is  tested  in  its  intended  environment  prior  to 
MS  C. 

In  ISD,  the  development  agency  conducts  DT&E  to  assist  with  engineering  design,  system  development, 
and  risk  identification  and  to  evaluate  the  contractors'  ability  to  attain  desired  technical  performance  in 
system  specifications  and  achieve  program  objectives.  The  DT&E  includes  T&E  of  components, 
subsystems,  and  prototype  development  models.  T&E  of  functional  compatibility,  interoperability,  and 
integration  with  fielded  and  developing  equipment  and  systems  is  also  included.  During  this  phase  of 
testing,  adequate  DT&E  is  accomplished  to  ensure  that  engineering  is  reasonably  complete  (including 
survivability/vulnerability,  compatibility,  transportability,  interoperability,  reliability,  maintainability, 
safety,  human  factors,  and  logistics  supportability).  Also,  this  phase  confirms  that  all  significant  design 
problems  have  been  identified  and  solutions  to  these  problems  are  in  place. 
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The  Service  OT&E  agency  should  conduct  an  OA  for  the  Post-Critical  Design  Review  Assessment  (PCDRA) 
to  estimate  the  systems  potential  to  be  operationally  effective  and  suitable;  identify  needed 
modifications;  and  provide  information  on  tactics,  doctrine,  organization,  and  personnel  requirements. 
The  early  OT&E  program  is  accomplished  in  an  environment  containing  limited  operational  realism. 
Typical  operational  and  support  personnel  are  used  to  obtain  early  estimates  of  the  user's  capability  to 
operate  and  maintain  the  system.  Some  of  the  most  important  products  of  user  assessments  of  system 
maintainability  and  supportability  are  human  factors  and  safety  issues. 

2. 2.4.2  T&E  Contributions  During  SC&MPD 

In  SC&MPD,  the  objective  is  to  design,  fabricate,  and  test  a  preproduction  system  that  closely 
approximates  the  final  product.  T&E  activities  of  the  EDM  during  this  period  yield  much  useful 
information.  For  example,  data  obtained  during  EDM  T&E  can  be  used  to  assist  in  evaluating  the  systems 
maintenance  training  requirements  and  the  proposed  training  program.  Test  results  generated  during 
EDM  T&E  also  support  the  user  in  refining  and  updating  employment  doctrine  and  tactics. 

T&E  activities  intensify  during  SC&MPD  and  make  significant  contributions  to  the  overall  acquisition 
decision  process.  During  SC&MPD,  T&E  is  conducted  to  satisfy  the  following  objectives: 

•  As  specified  in  program  documents,  assess  the  critical  technical  issues  by: 

o  Determining  how  well  the  development  contract  specifications  have  been  met. 

o  Identifying  system  technical  deficiencies  and  focus  on  areas  for  corrective  actions. 

o  Determining  whether  the  system  is  compatible  and  interoperable  and  can  be  integrated 
with  existing  and  planned  equipment  or  systems. 

o  Estimating  the  RAM  of  the  system. 

o  Determining  whether  the  system  is  safe  and  ready  for  LRIP. 

o  Evaluating  effects  on  performance  of  any  configuration  changes  caused  by  correcting 
deficiencies,  modifications,  or  product  improvements  (Pis). 

o  Assessing  human  systems  integration  (HSI)  and  identify  limiting  factors. 

•  Assess  the  technical  risk  and  evaluate  the  trade-offs  among  specifications,  operational 
requirements,  life  cycle  costs  (LCCs),  and  schedules. 

•  Assess  the  survivability,  vulnerability,  and  logistics  supportability  of  the  system. 

•  Verify  the  accuracy  and  completeness  of  the  technical  documentation  developed  to  maintain 
and  operate  the  weapons  system. 

•  Gather  information  for  training  programs  and  technical  training  materials  needed  to  support  the 
weapons  system. 
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•  Provide  information  on  environmental  issues  for  use  in  preparing  environmental  impact 
assessments. 

•  Determine  system  performance  limitations,  capabilities,  and  safe  operating  parameters. 

The  development  agency  evaluates  the  results  of  T&E  for  review  by  the  Service  headquarters  and  the 
Service  acquisition  review  council  prior  to  system  acquisition  review  by  the  MDA.  The  evaluation 
includes  the  results  of  testing  and  supporting  information,  conclusions,  and  recommendations  for 
further  engineering  development.  At  the  same  time,  the  OT&E  agency  prepares  an  independent  OA, 
which  contains  estimates  of  the  systems  potential  operational  effectiveness  and  suitability.  The  OA 
provides  a  permanent  record  of  OT&E  events,  an  audit  trail  of  OT&E  data,  test  results,  conclusions,  and 
recommendations.  This  information  is  used  to  prepare  for  MS  C  and  supports  a  recommendation  of 
whether  the  design  and  performance  of  the  system  in  development  justifies  proceeding  into  LRIP. 

2.2.5  P  &  D  Phase 

During  the  LRIP  work  effort,  the  purpose  is  to  achieve  an  operational  capability  that  satisfies  mission 
needs.  The  selected  system  design  and  its  principal  items  of  support  are  fabricated  as  production 
configuration  models.  Test  articles  normally  are  subjected  to  qualification  testing,  interoperability 
certification,  full-up  LET,  and  lOT&E.  This  work  effort  ends  with  the  FRPDR  or  a  Full-Rate  Production 
Deployment  Decision  Review  for  MAIS  or  software-intensive  systems  with  no  production  components. 
Timing  of  deployment  of  the  system  for  initial  operational  capability  (IOC)  is  usually  determined  by  the 
Service.  Key  documents  for  the  T&E  manager  at  the  time  of  the  FRPDR  are  the  updated  TEMP,  DT 
results,  the  Service  lOT&E  report,  and  LFT  report.  For  ACAT  I  and  designated  oversight  systems,  the 
DOT&E  is  required  by  law  to  document  the  assessment  of  the  adequacy  of  lOT&E  and  the  reported 
operational  effectiveness  and  suitability  of  the  system.  This  is  done  in  the  BLRIP  report.  Also  mandated 
by  law  is  the  requirement  for  the  DOT&E  to  submit  the  LFT  report  prior  to  the  program  proceeding 
beyond  LRIP.  These  DOT&E  reports  may  be  submitted  as  a  single  document. 

The  DOT&E  will  evaluate  whether  systems  are  production  representative.  Wherever  practicable,  lOT&E 
will  be  conducted  using  LRIP  systems  assembled  using  the  parts,  tools,  and  manufacturing  processes 
intended  for  use  in  full-rate  production  (FRP). 

The  system  will  also  utilize  the  intended  production  versions  of  software.  In  addition,  the  logistics 
system  and  maintenance  manuals  intended  for  use  on  the  fielded  system  should  be  in  place.  When  the 
use  of  LRIP  articles  is  impractical,  the  system  used  in  T&E  should,  at  a  minimum,  incorporate  the  same 
parts  and  software  to  be  used  in  LRIP  articles.  In  particular,  the  hardware  and  software  should  be  as 
defined  by  the  system-level  Critical  Design  Review  (CDR),  Functional  Configuration  Audit  (FCA),  and 
System  Verification  Review  (SVR),  including  correction  of  appropriate  major  deficiencies  identified 
during  DT.  Manufacturing  processes  to  be  used  in  FRP  should  also  be  adhered  to  as  closely  as  possible. 


32 


525  &-yR9l3&lzuSya  l-yt-B^SyuDcms 


/KI-L0J^b 


2.2.5.1  T&E  Contributions  Prior  to  FRPDR 

The  development  agency  transitions  the  final  design  to  LRIP  while  fixing  and  verifying  any  technical 
problems  discovered  during  the  final  testing  of  the  EDM  in  its  intended  environment.  The  maturity  of 
the  hardware  and  software  configurations  and  logistics  support  system  available  from  LRIP  are  assessed 
when  the  development  agency  considers  certifying  the  systems  readiness  for  lOT&E. 

The  DOT&E  must  be  provided  detailed  information  describing  any  process  differences  in  order  to 
independently  evaluate  whether  the  differences  are  acceptable.  The  Office  of  the  DOT&E  will  assess 
adherence  to  DoD  guidelines  as  part  of  its  responsibility  for  reviewing  and  approving  TEMPs  and  test 
plans.  Proposals  to  use  articles  that  are  not  from  LRIP  to  conduct  lOT&E  will  be  considered. 

lOT&E  is  conducted  prior  to  the  production  decision  at  FRPDR  to: 

•  Estimate  system  operational  effectiveness,  operational  suitability,  and  survivability. 

•  Identify  operational  deficiencies. 

•  Evaluate  changes  in  production  configuration. 

•  Provide  information  for  developing  and  refining  logistics  support  requirements  for  the  system 
and  training,  tactics,  techniques,  and  doctrine. 

•  Provide  information  to  refine  O&S  cost  estimates  and  identify  system  characteristics  or 
deficiencies  that  can  significantly  impact  O&S  costs. 

•  Determine  whether  the  technical  publications  and  support  equipment  are  adequate  in  the 
operational  environment. 

Before  the  certification  of  readiness  for  lOT&E,  the  Service  Acquisition  Executive  (SAE)  should  have 
obtained  the  JITC  certification  of  interoperability  for  the  system  components.  For  the  sake  of  efficiency 
and  completeness,  the  AOTR  should  be  conducted  in  conjunction  with  the  DoD  Components 
certification  of  readiness  review.  In  parallel  with  lOT&E,  LFT&E  may  be  used  to  evaluate  vulnerability  or 
lethality  of  a  weapon  system  as  appropriate  and  as  required  by  public  law.  The  PMs  briefing  and  the 
BLRIP  report  address  the  risks  of  proceeding  into  FRP. 

2. 2. 5.2  T&E  Contributions  After  the  FRPDR 

After  FRPDR,  when  the  FRP  decision  is  normally  made,  T&E  activities  continue  to  provide  important 
insights.  Tests  described  in  the  TEMP  but  not  conducted  during  earlier  phases  are  completed.  The 
residual  DT&E  may  include  extreme  weather  testing  and  testing  corrected  deficiencies.  System  elements 
are  integrated  into  the  final  operational  configuration,  and  DT  is  completed  when  all  system 
performance  requirements  are  met.  During  FRP,  Government  representatives  normally  monitor  or 
conduct  the  PAT&E.  Each  system  is  verified  by  PAT&E  for  compliance  with  the  requirements  and 
specifications  of  the  contract. 
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Post-production  testing  requirements  may  result  from  an  acquisition  strategy  calling  for  increment 
changes  to  accommodate  accumulated  engineering  changes  or  the  application  of  preplanned  product 
improvements  (P3ls).  Using  a  P3I  strategy  allows  parallel  development  of  new  technology  and  modular 
insertion  of  system  upgrades  into  production  equipment.  Technology  breakthroughs  and  significant 
threat  changes  may  require  system  modifications.  The  development  of  the  modifications  will  require  DT 
and  an  assessment  of  performance  change  to  determine  the  need  or  appropriate  level  of  OT. 

OT&E  activities  continue  after  the  FRP  decision  in  the  form  of  FOT&E.  The  initial  phase  of  FOT&E  may  be 
conducted  by  either  the  OT&E  agency  or  user  commands,  depending  on  Service  directives.  FOT&E 
verifies  the  operational  effectiveness  and  suitability  of  the  production  system,  determines  whether 
deficiencies  identified  during  lOT&E  have  been  corrected,  and  evaluates  areas  not  tested  during  lOT&E 
due  to  system  limitations.  Additional  FOT&E  may  be  conducted  over  the  life  of  the  system  to  refine 
doctrine,  tactics,  techniques,  and  training  programs  and  to  evaluate  future  increments,  modifications, 
and  upgrades. 

The  OT&E  agency  prepares  an  OA  report  at  the  conclusion  of  each  FOT&E.  This  report  records  test 
results,  describes  the  evaluation  accomplished  to  satisfy  critical  issues  and  objectives  established  for 
FOT&E,  and  documents  its  assessment  of  resolved  deficiencies.  Deficiencies  that  are  not  corrected  are 
recorded. 

A  final  report  on  FOT&E  may  also  be  prepared  by  the  using  command  test  team,  emphasizing  the 
operational  utility  of  the  system  when  operated,  maintained,  and  supported  by  operational  personnel 
using  the  concepts  specified  for  the  system.  Specific  attention  is  devoted  to  the  following: 

•  The  degree  to  which  the  system  accomplishes  its  missions  when  employed  by  operational 
personnel  in  a  realistic  scenario  (natural,  electronic,  threat)  with  the  appropriate  organization, 
doctrine,  supportability,  survivability,  vulnerability,  and  threat  (including  countermeasures; 
nuclear  threats;  effects;  and  nuclear,  biological,  and  chemical  (NBC))  environment  using  the 
tactics  and  techniques  developed  during  earlier  FOT&E. 

•  The  degree  to  which  the  system  can  be  placed  in  operational  field  use,  with  specific  evaluations 
of  availability,  compatibility,  transportability,  interoperability,  reliability,  wartime  usage  rates, 
maintainability,  safety,  human  factors,  manpower  supportability,  natural  environmental  effects 
and  impacts,  logistics  supportability,  and  documentation  and  training  requirements. 

•  The  conditions  under  which  the  system  was  tested  including  the  natural  weather  and  climatic 
conditions,  terrain  effects,  battlefield  disturbances,  and  enemy  threat  conditions. 

•  The  ability  of  the  system  to  perform  its  required  functions  for  the  duration  of  a  specified  mission 
profile. 

•  System  weaknesses  such  as  the  vulnerability  of  the  system  to  exploitation  by  countermeasures 
techniques  and  the  practicality  and  probability  of  an  adversary  exploiting  the  susceptibility  of  a 
system  in  combat. 
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A  specific  evaluation  of  the  personnel  and  logistics  changes  needed  for  the  effective  integration  of  the 
system  into  the  user's  inventory  is  also  made.  These  assessments  provide  essential  input  for  the  later 
acquisition  phases  of  the  system  development  cycle. 

2.2.6  O&S  Phase 

The  production  continues  at  full  rate  allowing  continued  deployment  of  the  system  to  operating 
locations  and  achievement  of  full  operational  capability  (FOC).  This  phase  may  include  major 
modifications  to  the  production  configuration,  increment  upgrades,  and  related  FOT&E.  Block  upgrades, 
P3I,  and  similar  efforts  that  provide  a  significant  increase  in  operational  capability  are  to  be  managed  as 
separate  increments  with  their  own  MS  B  and  MS  C  in  accordance  with  Reference  (d).  Evolutionary 
acquisition  permits  incremental  development  of  capability  and  modification,  but  it  is  still  necessary  to 
ensure  that  appropriate  planning  is  in  place  to  sustain  readiness  and  support  of  all  fielded  increments, 
and  for  some  major  modifications,  it  may  require  initiation  of  a  new  program.  To  determine  whether 
major  upgrades/modifications  are  necessary  or  deficiencies  warrant  consideration  of  replacement,  the 
MDA  may  review  the  impact  of  proposed  changes  on  system  operational  effectiveness,  suitability,  and 
readiness. 

2.3  T&E  and  SE  Processes 

T&E  activities  are  the  cornerstone  of  the  verification  and  validation  (V&V)  of  technical  processes  used  in 
SE,  as  well  as  supporting  technical  assessment  efforts  that  are  a  key  part  of  the  overall  DoD  SE  processes 
used  for  managing  systems  development.  NOTE:  DoD  systems  engineering  processes  are  derived  from 
IEEE  Std  15288-2008  (Reference  (v))  and  documented  in  Chapter  4  (Systems  Engineering)  of  Reference 
(j).  They  consist  of  technical  processes  used  for  systems  development  and  technical  management 
processes  used  for  controlling  development  efforts.  The  technical  processes  include  stakeholder 
requirements  definition,  requirements  analysis,  architectural  design,  implementation  (fabricate  or 
code),  integration,  verification,  validation,  and  transition.  The  technical  management  processes  include 
requirements  management,  interface  management,  risk  management,  configuration  management, 
technical  data  management,  technical  planning,  technical  assessment,  and  decision  analysis. 

Engineering  T&E  processes  are  a  significant  element  in  the  decision-making  process,  providing  data  that 
support  trade-off  analysis,  performance  verification,  risk  reduction,  and  requirements  refinement. 
Program  decisions  on  system  performance  maturity  and  readiness  to  advance  to  the  next  phase  of 
development  take  into  consideration  demonstrated  performance.  The  T&E  process  provides  data  that 
tell  the  developer  and  user  how  well  the  system  is  performing  during  development  and  if  it  is  ready  for 
fielding.  The  PM  must  balance  the  risks  of  cost,  schedule,  and  performance  to  keep  the  program  on 
track  to  production  and  fielding.  The  responsibility  of  decision-making  authorities  centers  on  assessing 
risk  trade-offs.  T&E  results  provide  key  information  in  this  regard. 

As  stated  in  Reference  (d) ,  T&E  shall  be  integrated  throughout  the  defense  acquisition  process.  T&E 
shall  be  structured  to  provide  essential  information  to  decision  makers,  assess  attainment  of  technical 
performance  parameters,  and  determine  whether  systems  are  operationally  effective,  suitable, 
survivable,  and  safe  for  intended  use.  The  conduct  of  T&E,  integrated  with  M&S,  shall  facilitate  learning. 
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assess  technology  maturity  and  interoperability,  facilitate  integration  into  fielded  forces,  and  confirm 
performance  against  documented  capability  needs  and  adversary  capabilities  as  described  in  the  STAR. 

SE  is  an  interdisciplinary  approach  to  evolve  and  verify  an  integrated  and  optimally  balanced  set  of 
product  and  process  designs  that  satisfy  user  needs  and  provide  information  for  management  decision 
making.  Some  of  the  systems  engineering  activities  for  one  phase  of  the  acquisition  life  cycle  are 
illustrated  in  Figure  2-3. 

A  systems  life  cycle  begins  with  the  user's  needs,  which  are  expressed  as  constraints,  and  the  required 
capabilities  needed  to  satisfy  mission  objectives.  Systems  engineering  is  essential  in  the  earliest  planning 
period,  in  conceiving  the  system  concept  and  defining  performance  requirements  for  system  elements. 
As  the  detailed  design  is  prepared,  systems  engineers  ensure  balanced  influence  of  all  required  design 
specialties,  including  testability.  Systems  engineers  resolve  interface  problems,  perform  design  reviews, 
perform  trade-off  analyses,  and  assist  in  verifying  performance. 

System  engineers  coordinate  the  many  specialized  engineers  involved  in  the  concurrent  engineering 
process  and  use  integrated  product  and  process  development  as  an  essential  tool.  IPTs  are  responsible 
for  the  integration  of  the  components  into  a  system. 

Through  interdisciplinary  integration,  a  systems  engineer  manages  the  development  efforts  and 
progress  of  product  definition  from  system  level  to  configuration-item  level,  detailed  level,  deficiency 
correction,  and  modifications/PIs.  Test  results  provide  feedback  to  analyze  the  design  progress  toward 
performance  goals.  Technical  management  processes  used  as  an  integral  part  of  systems  engineering 
include  technical  design  reviews,  configuration  management  (CM),  M&S,  TPM,  and  trade-off  analysis. 
Developmental  and  operational  testing  support  the  technical  reviews  by  providing  feedback  to  the 
systems  engineering  process.  More  information  on  the  reviews  is  contained  in  later  chapters. 

2.3.1  Importance  of  the  SEP 

Systems  engineering  is  the  iterative  logical  sequence  of  analysis,  design,  test,  and  decision  activities  that 
transforms  an  operational  need  into  the  descriptions  required  for  production  and  fielding  of  all 
operational  and  support  system  elements.  The  SEP  is  a  mandatory  program-unique  life-cycle  document 
that  describes  key  aspects  of  the  programs  systems  engineering  efforts.  The  SEP,  TEMP  and  other  T&E 
related  program  plans  should  be  aligned  to  ensure  that  they  are  mutually  supportive  and  consistent. 

2.3.2  Technical  Planning  Process  Activities 

The  technical  management  aspect  of  the  technical  planning  process  incorporates  management  planning 
for  the  integration  of  all  system  design  activities.  Its  purpose  is  to  develop  the  organizational 
mechanisms  and  document  them  in  various  plans,  for  direction  and  control,  and  identify  personnel  for 
the  attainment  of  program  objectives.  Technical  planning  defines  and  describes  the  type  and  degree  of 
systems  engineering  management,  the  SEP,  the  integration  of  several  related  program  plans,  and  the 
TEMP. 
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The  TEMP  must  be  consistent  with  other  program  technical  management  planning  efforts.  The  testing 
program  outlined  in  the  TEMP  must  ultimately  be  structured  so  as  to  provide  the  evaluation  data 
required  for  all  design  decision  points,  audits,  and  reviews  that  are  a  part  of  the  systems  engineering 
process.  A  supporting  process,  the  CM  process,  controls  the  systems  baseline  for  the  test  programs  to 
help  ensure  consistent  test  results  over  time  as  the  design  evolves  and  incorporates  design 
modifications  to  the  product  baseline  determined  to  be  necessary  by  T&E. 

The  TEMP  and  other  technical  management  planning  documents,  especially  the  SEP,  must  be  traceable 
to  each  other  and  consistent.  Key  functions  and  interfaces  of  the  system  with  other  systems  must  be 
described  and  correlated  with  the  systems  engineering  documentation  and  the  system  specification. 
Technical  thresholds  and  objectives  include  specific  performance  requirements  that  become  test 
planning  limits.  They  must  be  traceable  through  the  planned  systems  engineering  documentation  and 
can  be  correlated  to  the  content  of  the  measurement  program.  For  example,  failure  criteria  for 
reliability  thresholds  during  OT&E  must  be  delineated  and  agreed  upon  by  the  PM  and  the  operational 
test  director  (OTD)  and  reflected  in  the  TEMP. 

2.3.3  T&E  Role  in  Technical  Reviews  and  Audits 

The  role  of  T&E  changes  slightly  during  the  technical  reviews,  design  reviews,  physical  and  functional 
configuration  audits,  etc.,  that  are  performed  over  the  development  life  cycle  (see  Figure  2-4).  Typically, 
the  program  office  Chief  Developmental  Tester,  or  equivalent,  plans,  directs,  or  monitors  DT&E 
activities.  In  technical  reviews  and  audits,  the  role  of  the  Chief  Developmental  Tester  is  to  examine  the 
contractor's  approach  to  testing  and  evaluate  the  validity  of  the  process  and  the  accuracy  of  the 
contractor's  results  as  presented  during  the  reviews.  The  Chief  Developmental  Tester  uses  personal 
experience  and  background  in  T&E  to  assess  whether  the  contractor  did  enough  or  too  little  testing; 
whether  the  tests  were  biased  in  any  way;  and  whether  they  followed  a  logical  progression  using  the 
minimum  amount  of  time,  effort,  and  funds  while  still  gathering  and  providing  adequate  data  for  an 
informed  decision.  Each  type  of  technical  review  or  audit  will  have  a  different  focus/orientation  and 
specific  entry  and  exit  criteria,  but  the  Chief  Developmental  Tester  will  always  be  concerned  with  the 
testing  process  and  how  it  is  carried  out.  After  each  review,  the  Chief  Developmental  Tester  should 
always  document  all  observations  for  future  reference. 
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Figure  2-4  Illustrative  Systems  Engineering  Activities  (EMD  Phase) 

A  pictorial  roadmap  of  key  activities  in  the  systems  acquisition  processes-the  Integrated  Defense 
Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System  chart-is  available  at 
https://ilc.dau.mil .  The  short  title  for  this  chart  is  Integrated  Life  Cycle  Chart.  The  chart  is  a  training  aid 
for  DAU  courses  and  illustrates  the  interaction  of  the  three  key  processes  that  must  work  in  concert  to 
deliver  the  capabilities  required  by  the  Warfighters:  the  requirements  process  (Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  process);  the  acquisition  process  (Defense  Acquisition 
System  process);  and  the  program  and  budget  development  process  (Planning,  Programming, 
Budgeting,  and  Execution  (PPBE)  process). 

2.3.4  TPM 

TPM  identifies  critical  technical  parameters  that  are  at  a  higher  level  of  risk  during  design  and  forecasts 
their  future  values  over  time.  TPM  programs  track  T&E  data,  makes  predictions  about  whether  the 
parameter  can  achieve  final  technical  success  within  the  allocated  resources,  and  assists  in  managing 
the  technical  program. 

The  TPM  program  is  an  integral  part  of  the  T&E  program.  TPMs  programs  are  product  design 
assessments  and  are  part  of  the  backbone  of  the  DT  program.  TPMs  programs  estimate,  through 
engineering  analyses  and  tests,  the  future  values  of  essential  performance  parameters  of  the  current 
program  design. 
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2.3.5  System  Baselines  and  T&E 

The  SEP  describes  the  process  for  establishing  phase  technical  baselines  and  addresses  how  they  will  be 
controlled  throughout  the  acquisition  life  cycle.  These  baselines  (functional,  allocated,  product)  can  be 
modified  with  the  results  of  engineering  and  testing.  Management  of  such  changes  is  controlled  by  CM 
activities. 

An  essential  part  of  baseline  control  is  CM.  CM,  one  of  the  systems  engineering  technical  management 
processes,  benefits  the  T&E  community  in  three  ways:  (1)  the  baseline  to  be  used  for  testing  is 
determined;  (2)  changes  that  occur  to  the  baseline  as  a  result  of  testing  and  design  reviews  are 
incorporated  into  the  test  article  before  the  new  phase  of  testing  (to  prevent  retest  of  a  bad  design); 
and  (3)  CM  produces  statistically  significant  sample  sizes  that  give  greater  confidence  in  the  final  T&E 
results. 

2.3.6  T&E  in  support  of  Risk  Management 

Correcting  defects  later  in  the  system  development  life  cycle  has  been  estimated  to  add  from  10  percent 
to  30  percent  to  the  cost  of  each  item.  Such  costly  redesign  and  modification  efforts  can  be  reduced  if 
carefully  planned  and  executed  T&E  helps  to  detect  and  fix  system  deficiencies  early  in  the  acquisition 
process.  Fixes  instituted  during  early  work  efforts  cost  significantly  less  than  those  implemented  later, 
when  most  key  design  decisions  have  already  been  made. 

T&E  figures  prominently  in  the  decisions  reached  at  design  and  milestone  reviews.  However,  the  fact 
that  T&E  results  are  required  at  major  decision  points  does  not  presuppose  that  T&E  results  must  always 
be  favorable.  The  final  decision  responsibility  lies  with  the  decision  maker  who  must  examine  the  critical 
issues  and  weigh  the  facts.  Only  the  decision  maker  can  determine  the  weight  and  importance  that  is  to 
be  attributed  to  a  systems  capabilities  and  shortcomings  and  the  degree  of  acceptable  risk.  The 
decision-making  authority  will  be  unable  to  make  this  judgment  without  a  solid  base  of  information 
provided  by  T&E.  Figure  2-4  illustrates  the  LCC  of  a  system  and  how  the  timing  of  key  decisions  impacts 
program  expenditures. 
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PHASE:  “SA  tD  EMD  P&D  OPNS&  SUPPORT 


Figure  2-5  Impact  of  Decision  Timing  on  LCCs 
2.4  Five-Step  Illustrative  T&E  Process 

Outlined  below  and  in  Figure  2-6  is  an  illustrative  five-step  T&E  process.  This  process  can  provide 
answers  to  critical  T&E  questions  for  decision  makers  at  various  times  during  a  system  acquisition.  The 
T&E  process  should  begin  during  the  formative  stages  of  a  program  with  a  T&E  coordination  function,  in 
which  the  information  needs  of  the  various  decision  makers  are  formulated  in  conjunction  with  the 
development  of  the  program  requirements,  acquisition  strategy,  and  other  analyses. 

•  In  Step  1,  T&E  involvement  begins  with  early  feedback  on  measurability  and  testability  of  certain 
foundation  documents  that  describe  the  capability  needed.  Feedback  on  test  requirements  and 
resources  needed  to  support  critical  issue  evaluation  and  data  requirements  should  help  shape 
the  rationale  for  the  given  foundation  documents  that  will  baseline  Step  1  identification  of  T&E 
information  required  for  the  decision  maker.  The  required  information  usually  centers  on  the 
current  system  under  test,  which  may  be  in  the  form  of  concepts,  prototypes,  EDMs,  or 
production-representative/production  systems,  depending  on  the  acquisition  phase.  The 
required  information  consists  of  performance  evaluations  of  effectiveness  and  suitability, 
providing  insights  into  how  well  the  system  meets  the  user's  needs  at  a  point  in  time. 
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STEPS 


STEP  4 


MODELING  AND  SIMULATION 


Figure  2-6  Five  Step  Test  and  Evaluation  Process 

•  Step  2  is  the  pre-test  analysis  of  the  evaluation  objectives  from  Step  1  to  determine  the  types 
and  quantities  of  data  needed,  the  results  expected  or  anticipated  from  the  tests,  and  the 
analytical  tools  needed  to  conduct  the  tests  and  evaluations.  The  use  of  validated  models  and 
simulation  systems  during  pre-test  analysis  can  aid  in  determining  how  to  design  test  scenarios, 
how  to  set  up  the  test  environment,  how  to  properly  instrument  the  test,  how  to  staff  and 
control  test  resources,  how  best  to  sequence  the  test  trials,  and  how  to  estimate  outcomes. 

•  Step  3,  test  activity  and  data  management,  is  the  actual  test  activity  planning.  Tests  are 
conducted  and  data  management  for  data  requirements  is  identified  in  Step  2.  T&E  managers 
determine  what  valid  data  exist  in  historical  files  that  can  be  applied  and  what  new  data  must  be 
developed  through  testing.  The  necessary  tests  are  planned  and  executed  to  accumulate 
sufficient  data  to  support  analysis.  Data  are  screened  for  completeness,  accuracy,  and  validity 
before  being  used  for  Step  4. 

•  Step  4,  post -test  synthesis  and  evaluation,  is  the  comparison  of  the  measured  outcomes  (test 
data)  from  Step  3  with  the  expected  outcomes  from  Step  2,  tempered  with  technical  and 
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operational  judgment.  This  is  where  data  are  synthesized  into  information.  When  the  measured 
outcomes  differ  from  the  expected  outcomes,  the  test  conditions  and  procedures  must  be 
reexamined  to  determine  whether  the  performance  deviations  are  real  or  were  the  result  of 
test  conditions,  such  as  lack  of  fidelity  in  computer  simulation,  insufficient  or  incorrect  test 
support  assets,  instrumentation  error,  or  faulty  test  processes.  The  assumptions  of  tactics, 
operational  environment,  systems  performance  parameters,  and  logistics  support  must  have 
been  carefully  chosen,  fully  described,  and  documented  prior  to  test.  M&S  may  normally  be 
used  during  the  data  analysis  to  extend  the  evaluation  of  performance  effectiveness  and 
suitability. 

•  Step  5  is  when  the  decision  maker  weighs  the  T&E  information  against  other  programmatic 
information  to  decide  a  proper  course  of  action.  This  process  may  identify  additional 
requirements  for  test  data  and  iterate  the  DoD  T&E  process  again. 


2.5  Scientific  Test  and  Analysis  Techniques  (STAT) 

ST AT  implements  the  use  of  scientific  and  statistical  methods  in  developing  rigorous,  defensible  test 
plans  and  the  evaluation  of  their  results.  STAT  assists  the  test  community  in  improving  the  planning, 
execution,  analysis,  and  reporting  of  integrated  testing.  One  example  of  STAT  is  design  of  experiments 
(DOE). 

2.5.1  DOE 

DOE,  an  application  under  STAT,  is  a  structured  process  that  provides  the  scientific  and  statistical 
methods  needed  to  rigorously  plan  and  execute  testing.  By  following  this  process,  testers  gain  the 
following  benefits: 

•  An  understanding  of  the  likelihood  of  successfully  identifying,  as  well  as  the  likelihood  of 
mistakenly  identifying,  significant  performance  drivers. 

•  A  sound  method  for  determining  and  justifying  the  number  of  tests  planned  and  assets  needed 
to  obtain  required  information. 

•  The  ability  to  make  informed  trade-offs  of  test  costs  (assets)  versus  information  gained. 

•  A  rigorous  method  of  determining  specific  scenarios  that  provide  the  most  useful  data  for 
characterizing  system  performance  including  deficiencies  and  for  identifying  drivers  of  system 
performance. 

•  The  ability  to  identify  interactions  between  test  factors.  This  is  seldom  possible  with  other 
methods  of  developing  test  matrices. 

DOE  is  sequential  in  nature,  where  each  series  of  tests  informs  the  next  series  of  tests.  Evaluations 
should  take  into  account  all  available  and  relevant  data  and  information  from  contractor  test,  DT,  and 
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OT.  DT  results  will  be  used  to  support  OT  findings  and  scope  OT  events.  When  utilized  early  in  system 
development,  DOE  provides  the  most  efficient  method  of  identifying  system  deficiencies  so  that 
appropriate  and  timely  corrective  actions  can  be  taken  when  costs  of  correcting  deficiencies  are 
minimized. 

The  DOE  process  involves  the  following  steps: 

1.  Determine  the  purpose  of  the  test  to  include  output  variables  (results)  that  will  be  measured. 

2.  Determine  the  input  variables  (factors)  that  are  expected  to  have  an  impact  on  results. 

3.  Determine  the  levels  for  each  factor  (e.g.,  for  a  factor  range  to  target,  levels  may  be  2  km  and  10 
km). 

4.  Determine  constraints  that  must  be  considered  when  creating  the  test  matrix.  For  example,  a 
missile  fired  from  an  unmanned  aerial  vehicle  (UAV)  may  have  a  maximum  range  to  target  of  10 
km  from  a  UAV  at  high  altitude  but  only  a  5-km  range  from  low  altitude.  A  constraint  to  prevent 
tests  that  would  fire  a  missile  from  low  altitude  at  a  target  10  km  away  must  be  in  place  when 
creating  the  test  matrix. 

5.  Create  the  test  matrix  and  review  it  with  interested  parties/SMEs. 

6.  Execute  tests. 

7.  Analyze  test  data. 

8.  Make  decisions  based  on  analysis. 

In  reality,  steps  15  are  an  iterative  process  for  several  reasons  that  include  the  following: 

•  Constraints  are  not  always  easy  to  identify  and  SMEs  may  notice  impossible/impractical  test 
scenarios  in  a  matrix  that  were  not  identified  in  advance. 

•  The  number  of  test  assets  determined  to  be  necessary  may  not  be  affordable.  At  this  point, 
trade-offs  of  cost  vs.  information  must  be  made. 

Use  of  DOE  in  testing  is  appropriate  for  anything  other  than  a  simple  demonstration  of  capability.  It  is 
useful  in  planning  and  executing  component  tests,  simulation,  hardware-in-the-loop  (HWIL),  and  full- 
system  testing.  Ideally,  because  DOE  is  most  powerful  when  applied  sequentially,  DOE  will  be  utilized 
throughout  development  and  testing  in  all  areas.  For  example,  early  in  a  program,  digital  simulation  is 
often  the  most  mature  tool  available  for  evaluating  system  performance  and  making  design  decisions. 
DOE  is  used  to  create  simulation  test  matrices  to  identify  factors  that  should  be  tested  in  FIWIL.  A  DOE 
for  FIWIL  is  then  created  that  addresses  these  factors.  Then  data  from  FIWIL  is  used  to  modify  or  validate 
the  digital  simulation.  Because  DOE  was  used,  factors  for  which  results  in  digital  simulation  and  FIWIL 
are  different  can  be  identified  more  accurately  than  if  the  tests  were  planned  using  another  method. 
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DOE  is  a  topic  within  DAD  course  TST  203,  Intermediate  Test  and  Evaluation.  See  the  DAU  interactive 
catalog  of  courses  at  http://icatalog.dau.mil .  The  Services  also  conduct  courses  (from  basic  to 
advanced)  about  DOE;  the  command  or  Service  T&E  office  can  provide  more  details.  For  additional 
reading  and  reference  on  DOE,  many  books  are  commercially  available  online. 

During  the  approval  process  for  TEMPs  and  test  plans,  elements  of  experimental  design  and  the 
scientific  and  rigorous  approach  to  testing  will  be  reviewed.  The  TEMP  (i.e..  Parts  III  and  IV)  must  include 
this  information.  See  Reference  (d)  for  specific  guidance  on  required  TEMP  content  at  each  milestone. 

2.6  Summary 

T&E  is  an  essential  and  critical  discipline  and  an  engineering  tool.  T&E  is  used  to  identify  technical  risk, 
verify  performance,  and  validate  system  utility  throughout  the  defense  system  acquisition  life  cycle.  This 
iterative  cycle  consists  of  formally  defined  acquisition  phases  separated  by  discrete,  decision-point 
milestones. 

The  DoD  T&E  process  consists  of  developmental  and  operational  integrated  testing  that  is  used  to 
support  assessment  of  engineering  design  maturity  risk  and  programmatic  milestone  reviews.  The  T&E 
process  forms  an  essential  part  of  the  development  activities  used  by  system  developers  and  aids  in  the 
decision  process  used  by  senior  decision  authorities  in  DoD. 
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Test  &  Evaluation  Management  Guide 

Chapter  3  -  Program  Office  Responsibilities  for  T&E 

3.1  Introduction 

In  Government  acquisition  programs,  there  should  be  an  element  dedicated  to  the  management  of  T&E. 
This  element  has  the  overall  test  program  responsibility  for  all  phases  of  the  acquisition  process.  T&E 
expertise  may  be  available  through  matrix  support  or  reside  in  the  PMO  engineering  department. 

In  accordance  with  section  835  of  Public  Law  112-81  (Reference  (w)) ,  each  MDAP  will  be  supported  by  a 
chief  developmental  tester  and  a  governmental  test  agency  that  serves  as  the  lead  DT&E  organization 
for  the  program.  PMs  for  MDAP  and  MAIS  ACAT  I  and  lA  programs  shall  designate  a  chief  developmental 
tester  (i.e.,  T&E  key  leadership  position  (KLP))  who  will  be  responsible  for: 

Coordinating  the  planning,  management,  and  oversight  of  all  DT&E  activities  for  the  program. 

Maintaining  insight  into  contractor  activities  under  the  program  and  overseeing  the  T&E  activities  of 
other  participating  Government  activities  under  the  program. 

Helping  PMs  make  technically  informed  objective  judgments  about  contractor  DT&E  results  under  the 
program. 

PMs  for  all  programs  on  OSD  DT&E  oversight  shall  designate  a  Government  test  agency  to  serve  as  the 
lead  DT&E  organization  for  the  program.  The  lead  DT&E  organization  has  responsibility  for: 

Providing  technical  expertise  on  T&E  issues  to  the  chief  developmental  tester. 

Conducting  DT&E  activities  for  the  program,  as  directed  by  the  chief  developmental  tester. 

Assisting  the  chief  developmental  tester  in  providing  oversight  of  contractors  under  the  program. 

PMs  for  all  programs  should  establish  a  T&E  WIPT  to  include  SE,  DT&E,  OT&E,  and  representatives  of  all 
applicable  certification  authorities.  All  membership  designations  should  be  made  as  soon  as  practical 
after  the  Materiel  Development  Decision  (MDD)  and  should  be  maintained  until  the  program  is  removed 
from  OSD  T&E  oversight. 

The  T&E  WIPT  should  be  established  and  chartered  as  early  as  possible  during  the  MSA  phase.  The  T&E 
WIPT  is  a  defined  forum  supporting  the  PM/PMO  as  the  T&E  SMEs  responsible  for  all  aspects  of  a 
programs  T&E  effort.  This  effort  includes  T&E  program  strategy,  design,  development,  oversight, 
support  to  other  program  WIPTs,  and  the  analysis  assessment  and  reporting  of  test  results.  Membership 
consists  of  the  PM  (Chair),  Chief  Developmental  Tester  (the  PM  may  designate  the  Chief  Developmental 
Tester  as  Chair  (for  ACAT  I  and  ACAT  lA  programs,  the  Chief  Developmental  Tester  is  the  chief 
developmental  tester)),  OSD  (DASD(DT&E)  and  DOT&E),  invited  SMEs,  functional  representatives, 
certifying  agencies,  and  contractors.  The  PM  may  form  lower  level  functional  working  groups,  which 
report  to  the  T&E  WIPT,  to  focus  on  specific  areas  such  as  integrated  test  planning,  lA,  software  T&E, 
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reliability,  M&S  development  and  verification,  validation,  and  accreditation  (VV&A),  and  threat  support. 
The  T&E  WIPT  is  established  to: 

•  Provide  a  forum  for  all  key  organizations  to  be  involved  in  the  T&E  effort. 

•  Develop  and  maintain  the  program  T&E  strategy. 

•  Develop  and  manage  an  integrated  test  program  for  M&S,  DTs,  and  OTs  to  support  evaluations. 

•  Develop  and  maintain  the  integrated  test  schedule. 

•  Identify  and  resolve  issues. 

Document  a  quality  TES/TEMP  schedule  as  quickly  and  as  efficiently  as  possible,  which  necessitates  that 
all  interested  parties  are  afforded  an  opportunity  to  contribute  to  the  TES/TEMP  development. 

Typically,  a  Chief  Developmental  Tester  position  is  established  in  a  program  office  to  act  as  the  focal 
point  for  all  T&E  efforts.  The  Chief  Developmental  Tester  normally  leads  the  T&E  WIPT,  uses  the  SMEs  to 
accomplish  all  tasks  related  to  T&E,  and  consults  with  other  T&E  stakeholders  on  the  T&E  WIPT  to  obtain 
correct,  accurate,  and  up-to-date  information  on  T&E  best  practices.  The  Chief  Developmental  Tester 
uses  the  T&E  WIPT  to  accomplish  the  T&E  mission  and  carry  out  the  responsibilities  of  the  position.  The 
five  blocks  in  Figure  3-1  list  the  major  T&E  items  or  areas  for  each  of  the  five  acquisition  phases  in  which 
a  T&E  WIPT  must  be  involved.  The  items  listed  are  not  the  only  items  of  T&E  WIPT  concern.  Items  may 
be  added  depending  on  the  nature  of  the  product/system  under  development.  For  example,  if  the 
product/system  is  for  multi-Service/Defense  Agency  use,  there  may  be  memorandums  of  agreement 
(MOAs)  and/or  memorandums  of  understanding  (MOUs)  regarding  integrated  testing,  evaluation,  and 
assessments. 

Ideally,  the  PMO  engages  a  T&E  SME  during  the  MSA  phase  to  assist  in  early  assessments  of  materiel 
solutions  and  development  of  test  strategies  that  will  support  the  acquisition  strategy  at  MS  A.  By  the 
start  of  the  TD  phase,  a  PMO  should  have  a  Chief  Developmental  Tester.  In  the  PMO,  the  Chief 
Developmental  Tester  would  be  responsible  for  defining  the  scope  and  concept  of  the  test  program, 
establishing  the  overall  program  test  objectives,  and  managing  test  program  funds  and  coordination. 

The  Chief  Developmental  Tester  should  develop  the  RFP  language  regarding  testing,  in  close 
coordination  with  systems  engineering,  RAM,  and  logistics  activities 
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Figure  3-1  T&E  WIPT  Focus  Areas 

and  closely  coordinate  schedule  and  resources  with  his  or  her  logistics  (training,  logistics 
demonstrations)  and  RAM  counterparts.  The  Chief  Developmental  Tester  will  typically  be  responsible  for 
setting  up  data  collection  at  various  field  activities  that  will  support  RAM,  logistics,  and  quality  team 
members,  as  well  as  allocating  test  assets  to  support  various  activities.  The  Chief  Developmental  Tester 
should  provide  test  directors  for  detailed  planning  of  DT  phases  and  test  events  as  required  and 
coordinate  test  resources,  facilities,  and  their  support  required  for  each  phase  of  testing.  In  addition,  the 
Chief  Developmental  Tester  or  a  staff  member  will  be  responsible  for  managing  the  TEMP  and  planning 
and  managing  any  special  test  programs  required  for  the  program.  The  Chief  Developmental  Tester  will 
also  review,  evaluate,  approve,  and  release  for  distribution  contractor-prepared  test  plans  and  reports, 
and  review  and  coordinate  all  appropriate  Government  test  plans.  The  Chief  Developmental  Tester 
prepares  criteria,  coordinates  test  readiness  reviews  (TRRs),  and  participates  in  systems  engineering 
technical  reviews  (SETRs)  and  milestone  reviews  by  providing  results  of  test  reports  and  relevant 
information  on  updates  to  future  testing  based  on  current  system  development  progress.  After  the 
system  is  produced,  the  Chief  Developmental  Tester  will  be  responsible  for  supporting  PAT&E  and  the 
test  portions  of  later  increments,  upgrades,  or  enhancements  to  the  weapon  system/acquisition. 

3.2  Relationship  To  The  PM 

The  PM  is  ultimately  responsible  for  all  aspects  of  system  development,  including  testing.  The  Chief 
Developmental  Tester,  a  highly  experienced  T&E  professional,  is  normally  authorized  by  the  PM  to 
conduct  all  duties  in  the  area  of  T&E.  Inputs  from  the  Chief  Developmental  Tester  to  the  contract, 
engineering  specifications,  systems  engineering  efforts,  budget,  program  schedule,  etc.,  are  essential  if 
the  PM  is  to  manage  T&E  aspects  of  the  program  efficiently.  The  Chief  Developmental  Tester  requires 
strong  personal  communication  and  teaming  skills  to  ensure  that  independent  testing  and  reporting 
effectively  contribute  to  the  overall  goals  that  the  PM  establishes  for  cost,  schedule,  and  performance. 

In  accordance  with  the  USD(AT&L)  Memorandum  (Reference  (x)),  the  Program  Lead  Test  and  Evaluation 
(chief  developmental  tester)  for  ACAT  I  and  lA  programs  shall  be  designated  as  the  T&E  KLP.  This  section 
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contains  typical  suggested  activities  that  such  an  individual  would  accomplish  in  support  of  the  PM 
mission. 

Before  the  initial  draft  of  the  RFP  is  created,  the  Chief  Developmental  Tester  must  form  an  IPT,  a  test 
planning/integration  working  group.  This  group  should  include  members  from  the  OTA,  DT  agency, 
organizations  that  may  be  jointly  acquiring  the  same  system,  test  supporting  agencies,  operational 
users,  oversight  authorities  (e.g.,  DASD(DT&E),  DOT&E),  and  any  other  organizations  that  will  be 
involved  in  the  test  program  by  providing  test  support  or  by  conducting,  evaluating,  or  reporting  on 
testing.  The  Chief  Developmental  Tester  should  stand  up  the  T&E  WIPT  prior  to  the  development  of  the 
RFP,  if  possible.  When  a  program  has  joint  interoperability  requirements,  a  DISA  capability  test  team 
(CTT)  chairman  represents  the  DISA  commander  and  JITC  at  program  T&E  WIPT  and  similar  working 
groups.  The  functions  of  the  groups  are  to  facilitate  the  use  of  testing  expertise,  instrumentation, 
facilities,  simulations  and  models;  integrate  test  requirements;  accelerate  the  TES/TEMP  coordination 
process;  resolve  test  cost  and  scheduling  problems;  implement  a  STAT  process  that  maximizes  the  use  of 
data  to  increase  the  use  of  scientific  and  statistical  methods  in  developing  rigorous,  defensible  test  plans 
and  in  evaluating  their  results;  and  provide  a  forum  to  ensure  that  T&E  of  the  system  is  coordinated.  The 
existence  of  a  test  coordinating  group  does  not  alter  the  responsibilities  of  any  command  or 
headquarters;  and  in  the  event  of  disagreement  within  a  group,  the  issue  is  resolved  through  the  normal 
command/staff  channels.  In  later  meetings,  the  contractor  participates  in  this  test  planning  group; 
however,  the  contractor  may  not  be  selected  by  the  time  the  first  meetings  are  held. 

The  purpose  of  these  meetings  is  to  develop  early  test  documentation,  the  TES  documents,  and  to  agree 
on  basic  test  program  schedules,  scope,  support,  etc.  The  TEMP  serves  as  the  top-level  test 
management  document  for  the  acquisition  program,  requiring  updates  when  necessary  and  at  specific 
milestones. 

3.3  Early  Program  Stages 

In  the  early  stages  of  a  program,  the  T&E  function  is  often  handled  by  matrix  support  from  the 
sponsoring  materiel  command.  Matrix  T&E  support  or  the  Chief  Developmental  Tester  should  be 
responsible  for  development  of  T&E  plans  and  planning  documents  and  the  T&E  sections  of  the  RFP  in 
accordance  with  the  Office  of  the  DASD(DT&E)  Guidebook  (Reference  (y))  for  developing  sections  of  the 
RFP.  The  ultimate  responsibility  for  the  RFP  is  between  the  PM  and  the  primary  contracting  officer 
(PCO),  however,  the  Chief  Developmental  Tester  is  responsible  for  several  areas,  for  example:  the  test 
schedule,  test  program  funding  (projections),  test  data  requirements  for  the  program  (test  reports, 
plans,  procedures,  quick-look  reports,  etc.),  determination  of  appropriate  verification  methods  and  their 
documentation  in  part  4  of  the  systems  specification,  coordination  with  logistics  and  RAM  regarding  test 
assets  and  resources  needed  to  support  their  efforts,  the  test  section  of  the  statement  of  work  (SOW), 
portions  of  the  acquisition  plan,  information  for  proposal  preparation,  feedback  on  the  CDD,  and  if  a 
joint  acquisition  program,  integration  of  multiple  Services  requirements. 
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3.3.1  Role  of  Memorandums 

Early  in  the  program,  another  task  of  the  Chief  Developmental  Tester  is  the  arrangement  of  any 
MOAs/MOUs  or  program-level  agreements  between  the  Services,  North  Atlantic  Treaty  Organization 
(NATO)  countries,  test  organizations,  etc.,  which  outline  the  responsibilities  of  each  organization.  The 
RFP  and  SOW  outline  contractor/Government  obligations  and  arrangements  on  the  access  and  use  of 
test  facilities  (contractor  or  Government  owned). 

3.3.2  Test  Data  Management 

The  Chief  Developmental  Tester  will  have  approval  authority  for  selected  contractor-created  test  plans, 
procedures,  and  reports.  The  Chief  Developmental  Tester  must  have  access  to  selected  contractor 
testing  and  test  results,  and  the  Chief  Developmental  Tester  is  responsible  for  disseminating  the  results 
to  Government  agencies  that  need  this  data.  The  DASD(DT&E),  the  DOT&E,  and  their  designated 
representatives  shall  have  full  and  immediate  access  to  all  records,  reports,  and  data  including  but  not 
limited  to  data  from  all  tests,  system  logs,  execution  logs,  test  director  notes,  user/operator 
assessments,  etc.  All  data  include  but  are  not  limited  to  classified,  unclassified,  and  competition- 
sensitive  data,  or  proprietary  data  when  available.  Data  may  be  preliminary  and  shall  be  identified  as 
such.  Additionally,  the  Chief  Developmental  Tester  creates  report  formats  and  timelines  for  contractor 
submittal.  Government  approval,  etc.  (  Reference  (d) ) 

The  data  requirements  for  the  entire  test  program  are  outlined  in  the  contract  data  requirements  list 
(CDRL).  The  Chief  Developmental  Tester  should  review  the  Acquisition  Streamlining  &  Standardization 
Information  System  (ASSIST)  which  is  now  the  official  source  of  all  documents  listed  in  the  DoDISS  and 
all  DIDS.  The  ASSIST  database  is  maintained  by  the  Defense  Automated  Printing  Service  (DAPS)  in 
Philadelphia,  PA.  A  feature  called  "QuickSearch"  lets  users  search  the  ASSIST  database  for  defense  and 
federal  specifications  and  standards,  military  handbooks.  Qualified  Products  Lists  (QPLs),  and  other 
documents  listed  in  the  DoDISS  and  DIDS.  "Assist  Online"  offers  the  features  of  QuickSearch,  plus 
analyses,  reports,  SD-1  contracts,  project  and  other  information.  New  users  must  complete  the  online 
registration.  Log  In  Accounts  for  the  ASSIST  are  free  to  DoD  and  to  the  public.  The  address  to  ASSIST  is 
https://assist.daps.dla.mil .  The  Chief  Developmental  Tester  provides  input  to  this  section  of  the  RFP 
early  in  the  program.  The  Chief  Developmental  Tester  ensures  that  the  office  and  all  associated  test 
organizations  requiring  the  information  receive  the  required  test  documentation  on  time.  Usually,  the 
contractor  sends  the  data  packages  directly  to  the  Chief  Developmental  Tester,  who  in  turn  has  a 
distribution  list  trimmed  to  the  minimum  number  of  copies  for  agencies  needing  that  information  to 
perform  their  mission  and  oversight  responsibilities.  It  is  important  for  the  Chief  Developmental  Tester 
to  use  an  integrated  test  program  and  request  contractor  test  plans  and  procedures  well  in  advance  of 
the  actual  test  performance  to  ensure  that  the  office  of  the  Chief  Developmental  Tester  has  time  to 
approve  the  procedures  or  implement  modifications. 

The  Chief  Developmental  Tester  must  receive  the  test  results  and  reports  on  time  to  enable  the  office  of 
the  Chief  Developmental  Tester,  the  PM,  and  higher  authorities  to  make  program  decisions.  The  data 
received  should  be  tailored  to  provide  the  minimum  information  needed.  The  Chief  Developmental 
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Tester  must  be  aware  that  data  requirements  in  excess  of  the  minimum  needed  may  lead  to  an 
unnecessary  increase  in  overall  program  cost.  For  data  that  are  needed  quickly  and  informally  (at  least 
initially),  the  Chief  Developmental  Tester  can  request  quick-look  reports  that  give  test  results 
immediately  after  test  performance.  The  Chief  Developmental  Tester  is  also  responsible  for  coordinating 
with  the  contractor  on  all  report  formats  and  the  specific  format  requirements  in  the  contract.  In 
general,  the  in-house  contractor  format  is  acceptable  in  most  cases  and  the  Chief  Developmental  Tester 
must  determine  the  reports  utility  before  adding  cost  to  contracts  for  using  other  specific  T&E  formats. 

The  contract  must  specify  what  data  the  contractor  will  supply,  which  organizations  will  receive  the 
data,  and  whether  any  outside  agencies  will  be  included  in  direct  distribution.  The  PMO  Chief 
Developmental  Tester  should  include  the  OTA  on  the  distribution  list  for  all  test  documents  that  are  of 
concern  during  the  DT&E  phase  of  testing  so  they  will  be  informed  of  test  item  progress  and  previous 
testing.  In  this  way,  the  OTA  will  be  informed  when  developing  its  own  test  plans  and  procedures  for 
OT&E.  In  fact,  OTA  representatives  should  attend  the  CDRL  Review  Board  meeting  and  provide  the  PMO 
with  a  list  of  the  types  of  documents  the  OTA  will  need.  The  Chief  Developmental  Tester  should 
coordinate  the  test  sections  of  this  data  list  with  the  OTA  and  indicate  concerns  at  that  meeting.  All 
contractor  test  reports  should  be  made  available  to  the  OTA.  In  return,  the  Chief  Developmental  Tester 
must  stay  informed  of  all  OTA  activities,  understand  the  test  procedures,  and  plan  and  receive  the  test 
reports.  Unlike  DT&E  and  contractor  documentation,  the  PMO  Chief  Developmental  Tester  will  not  have 
report  or  document  approval  authority  for  OT&E  items.  The  Chief  Developmental  Tester  is  responsible 
for  keeping  the  PM  informed  of  OT&E  results.  Additional  information  is  available  in  Reference  (y) . 

3.3.3  Test  Schedule  Formulation 

An  important  task  the  Chief  Developmental  Tester  has  during  the  creation  of  the  RFP  is  a  test  program 
schedule  that  supports  the  evaluation  framework  (contractor  and  Government  DT&E  plus  OT&E  events). 
Initially,  the  PM  will  need  contractor  predictions  of  the  hardware  availability  dates  for  models, 
prototypes,  mock-ups,  full-scale  models,  etc.,  once  the  contract  is  awarded.  The  Chief  Developmental 
Tester  uses  this  information  and  the  early  test  planning  in  the  TES  to  create  a  realistic  front-end 
schedule  of  the  in-house  testing  the  contractor  will  conduct  before  Government  testing  (DT&E,  OT&E, 
and  interoperability,  lA,  and  security  testing).  Then,  a  draft  integrated  schedule  is  developed  upon  which 
the  Government  testing  schedules  can  be  formulated  and  contractor  support  requirements  can  be 
determined.  The  Chief  Developmental  Tester  can  use  past  experience  in  testing  similar  weapon 
systems/acquisition  items  or  contract  test  organizations  that  have  the  required  experience  to  complete 
the  entire  test  schedule.  Because  the  test  schedule  is  a  critical  contractual  item,  contractor  input  is  very 
important.  The  test  schedule  will  normally  become  an  item  for  negotiation  once  the  RFP  is  released  and 
the  contractor's  proposal  is  received.  Attention  must  be  given  to  ensure  that  the  test  schedule  is  not  so 
success  oriented  that  retesting  of  failures  will  cause  serious  program  delays  for  either  the  Government 
test  agencies  or  the  contractor. 

Another  important  early  activity  to  be  performed  by  the  Chief  Developmental  Tester  is  to  coordinate  the 
OT&E  test  schedule.  Since  the  contractor  may  be  required  to  provide  support,  the  OT&E  test  support 
may  need  to  be  contractually  agreed  upon  before  contract  award.  Sometimes,  the  Chief  Developmental 


50 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcmS 


/Kl-L0i, 


Tester  can  formulate  a  straw-man  schedule  (based  on  previous  experience)  and  present  this  schedule  to 
the  OT  representative  at  the  initial  T&E  IPT  meeting  for  review,  or  the  Chief  Developmental  Tester  can 
contact  the  OTA  and  arrange  a  meeting  to  discuss  the  new  program.  In  the  meeting,  time  requirements 
envisioned  by  the  OTA  can  be  discussed.  Input  from  that  meeting  then  goes  into  the  RFP  and  to  the  PM. 

Before  setup  of  lOT&E,  certification  of  readiness  for  lOT&E  may  require  a  time  gap  for  review  of  DT&E 
test  results  and  refurbishment  or  corrections  of  deficiencies  discovered  during  DT&E.  Time  is  also 
needed  for  the  OTA  to  make  final  arrangements  to  start  its  portion  of  the  T&E  program.  The  test 
schedule  for  DT&E  should  not  be  overrun  so  that  the  lOT&E  test  schedule  is  adversely  impacted, 
reducing  program  schedule  time  with  inadequate  OT  or  rushing  the  reporting  of  lOT&E  results.  For 
example,  if  the  DT&E  schedule  slips  6  months,  the  OT&E  schedule  and  milestone  decision  should  slip 
also.  lOT&E  should  not  be  shortened  just  to  make  a  milestone  decision  date. 

3.3.4  Programmatic  Environmental  Analysis 

PMO  personnel  should  be  sensitive  to  the  potential  environmental  consequences  of  system  materials, 
operations,  and  disposal  requirements.  Public  Laws  (parts  15001508  of  Title  40,  Code  of  Federal 
Regulations  (Reference  (z));  National  Environmental  Policy  Act  (NEPA)  regulations;  Executive  Order 
12114  (Reference  (aa));  DoD  5000  Series;  etc.)  require  analysis  of  hazardous  materials  and  appropriate 
mitigation  measures  during  each  acquisition  phase.  As  indicated  in  Reference  (j),  planning  should 
consider  the  potential  testing  impacts  on  the  environment. 

Litigation  resulting  in  personal  fines  and  imprisonment  against  Government  employees  has  raised  the 
environmental  awareness  at  test  ranges  and  facilities.  Environmental  impact  statements  (EISs) 
(supported  by  long,  thorough,  and  costly  studies  and  public  testimony)  or  environmental  analysis  and 
assessments  are  generally  required  before  any  system  testing  can  be  initiated. 

3.4  PMO/ContractorTest  Management 

The  PMO  will,  in  most  cases,  have  a  contractor  test  section  counterpart.  With  this  counterpart,  the  Chief 
Developmental  Tester  works  out  the  detailed  test  planning,  creation  of  schedules,  etc.,  for  the  system 
end-to-end  test  program.  The  PMO  uses  input  from  all  sources  (contracts,  DT  agencies,  OTAs,  higher 
headquarters,  etc.)  to  formulate  the  test  programs  length,  scope,  and  necessary  details.  The  Chief 
Developmental  Tester  ensures  that  the  RFP: 

Describes  exactly  how  the  contractor  will  support  Government  T&E  and  the  contractor's  role  in  the 
acquisition  and  T&E  processes. 

Requires  Government  review  of  contractor  test  plans  before  test  execution  and  that  the  contractor's 
deficiency  reporting  (DR)  system  is  compatible  with  and  feeds  into  the  Governments  DR  system. 

Includes  provisions  for  Government  attendance  at  contractors'  tests  and  that  contractor  test  results, 
consistent  with  the  contract,  are  provided  to  the  Government. 

See  Reference  (y)  for  additional  guidance. 
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Experience  indicates  that  most  programs  use  a  success  based  timeline  when  planning  the  integrated 
program  schedule,  meaning  that  each  event  or  activity  is  based  on  positive  results  and  moving  to  the 
next  activity  or  phase  of  the  acquisition  effort.  Experience  also  indicates  that  this  concept  is  a  major 
fault  in  most  program  planning.  Although  T&E  is  best  managed  as  event  driven,  in  most  cases  it  is  not 
practical  in  practice;  therefore,  the  Chief  Developmental  Tester  must  provide  realistic  timelines  for 
preparation,  execution,  analysis,  and  reporting  to  enable  the  PM  to  effectively  balance  cost,  schedule, 
and  performance.  The  most  significant  impact  of  success-based  planning  is  the  lOT&E.  This  is  a 
statutorily  mandated  requirement  that  comes  near  the  end  of  the  development  cycle,  and  meant  to 
provide  the  Head  of  the  Component  and  MDA  with  an  independent  evaluation  of  the  operational 
effectiveness,  operational  suitability,  and  survivability  of  the  system.  The  Chief  Developmental  Tester 
must  keep  the  PMO  informed  of  pending  risks  to  the  adequacy  of  schedule  in  completing  post-DT&E 
analysis,  reporting,  and  system  corrections  before  proceeding  into  lOT&E  with  its  subsequent  timeline 
for  preparations,  execution,  analysis,  and  reporting  before  the  milestone  date.  Major  delays  to  any 
testing  schedule  require  full  consideration  of  the  PMO  and  MDA  in  slipping  milestone  decisions  a 
corresponding  amount  of  time. 

Once  agreement  on  the  contractor's  technical  approach  is  reached,  the  Chief  Developmental  Tester  is 
responsible  for  providing  inputs  to  the  Government  contracting  officer  during  negotiations.  The  T&E 
contract  deliverables  are  assigned  contract  line  item  numbers,  which  are  created  by  the  Chief 
Developmental  Tester.  This  assignment  will  ensure  that  the  contractor  delivers  the  required 
performances  at  specified  intervals  during  the  life  of  the  contract.  Usually,  there  will  be  separate 
contracts  for  development  and  production  of  the  acquisition  item.  For  each  type  of  contract,  the  Chief 
Developmental  Tester  is  responsible  for  providing  the  PCO  and  PM  with  T&E  input. 

For  programs  on  DT&E  oversight,  the  PM  must  identify  each  test  phase  or  major  event  in  the  TEMP  as 
contractor  or  Government  DT&E.  All  programs  must  plan  for  the  conduct  of  DT&E  or  integrated  testing 
that  includes  or  is  led  by  Government  personnel,  so  as  to  provide  confidence  that  the  system  design 
solution  is  on  track  to  execute  the  operational  scenarios  identified  by  the  OTAs. 

3.5  Test  Program  Funding/Budgeting 

The  PMO  must  identify  funds  for  testing  early  so  that  test  resources  can  be  obtained.  The  Chief 
Developmental  Tester  uses  the  acquisition  schedule,  TEMP,  and  other  program  and  test  documentation 
to  identify  test  resource  requirements.  The  Chief  Developmental  Tester  coordinates  these  requirements 
with  the  contractor  and  Government  organizations  that  have  the  test  facilities  to  ensure  that  the 
facilities  are  available  for  testing.  The  Chief  Developmental  Tester  ensures  that  test  costs  include 
contractor  and  Government  test  costs.  The  contractor's  test  costs  are  normally  outlined  adequately  in 
the  proposal;  however,  the  Government  test  ranges,  instrumentation,  and  test-support  resource  costs 
must  be  determined  by  other  means.  Usually,  the  Chief  Developmental  Tester  contacts  the  test 
organization  and  outlines  the  test  program  requirements,  and  the  test  organization  sends  the  program 
office  an  estimate  of  the  test  program  costs.  The  Chief  Developmental  Tester  then  obtains  cost 
estimates  from  all  test  sources  that  the  Chief  Developmental  Tester  anticipates  using  and  supplies  this 
information  to  the  PM.  Programs  should  use  DoD  Government  T&E  capabilities  and  invest  in 
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Government  T&E  infrastructure  unless  an  exception  can  be  justified  as  cost-effective  to  the 
Government.  The  Chief  Developmental  Tester  should  also  ensure  that  any  program  funding  reductions 
are  not  absorbed  entirely  by  the  test  program.  Some  cutbacks  may  be  necessary  and  allowable,  but  the 
test  program  should  supply  decision  makers  with  enough  information  to  make  rational  program 
milestone  decisions. 

3.6  Contractor  Testing 

The  Chief  Developmental  Tester  is  responsible  for  ensuring  that  contractor-conducted  tests  are 
monitored  by  the  Government  to  allow  for  evaluation  of  the  design  solution.  The  Chief  Developmental 
Tester  must  be  given  access  to  appropriate  contractor  data,  test  results,  and  test  reports,  consistent 
with  the  technical  data  rights  and  access  provisions  that  have  been  contractually  specified.  The  Chief 
Developmental  Tester  should  be  judicious  in  deciding  whether  to  require  Government  approval  of 
contractor  test  plans  and  whether  the  Government  has  the  right  to  witness  all  contractor-run  test 
events,  take  delivery  of  all  contractor  test  data,  and  approve  contractor  test  reports.  All  such 
Government  involvement  adds  cost  and  schedule  to  the  contractor  test  program  and  should  be  carefully 
planned  and  managed. 

Usually,  the  contract  requires  that  Government  representatives  be  informed  ahead  of  time  of  any 
testing  the  contractor  conducts  so  that  the  Government  can  arrange  to  witness  certain  testing  or  receive 
results  of  the  tests.  Further,  the  contractor's  internal  data  should  be  available  as  a  contract  provision. 
The  Chief  Developmental  Tester  must  ensure  that  Government  test  personnel  (DT&E/OT&E)  have  access 
to  contractor  test  results.  It  would  be  desirable  to  have  all  testers  observe  some  contractor  tests  to  help 
develop  confidence  in  the  results  and  identify  areas  of  risk.  NOTE:  The  Defense  Contract  Management 
Agency  (DCMA),  as  part  of  its  mission,  has  in-plant  representatives  at  various  defense  manufacturing 
facilities.  As  part  of  their  support  to  the  program  office,  a  formal  MOA  is  negotiated  with  the  PMO.  That 
MOA  may  include  support  from  DCMA  to  witness  various  in-plant  tests,  which  can  be  an  essential 
resource  in  this  area.  Additional  information  is  available  in  Reference  (y). 

3.7  Specifications 

Within  the  program  office,  the  engineering  section  usually  is  responsible  for  the  system  performance 
specification(s)  as  released  with  the  RFP.  The  contractor  is  then  tasked  with  creating  the  detailed  design 
specification  documentation  as  called  out  by  the  contract,  which  will  be  delivered  once  the  item/system 
design  is  formalized  for  production. 

The  Chief  Developmental  Tester  performs  an  important  function  in  specification  formulation  by 
reviewing  the  specifications  to  determine  whether  performance  parameters  and  stated  capability  needs 
are  testable;  whether  current,  state-of-the-threat  technology  can  determine  (during  the  DT&E  test 
phase)  whether  the  performance  specifications  are  being  met  by  the  acquisition  item;  or  whether  the 
specified  parameters  are  too  tight.  A  specification  is  too  tight  if  the  requirements  (section  3)  are 
impossible  to  meet  or  demonstrate,  or  if  the  specification  has  no  impact  on  the  form,  fit,  or  function  of 
the  end-item,  the  system  of  which  it  will  become  a  part,  or  the  system  with  which  it  will  interact.  The 
Chief  Developmental  Tester  must  determine  whether  test  objectives  can  be  adequately  formulated  from 
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those  specifications  that  will  provide  thresholds  of  performance,  minimum  and  maximum  standards, 
and  reasonable  operating  conditions  for  the  end-items  final  mission  and  operating  environment.  The 
specifications  shape  the  DT&E  scenario,  test  ranges,  test  support,  targets,  etc.,  and  the  way  in  which  the 
specifications  are  developed  and  stated  is  very  important  to  the  Chief  Developmental  Tester. 

Among  other  quality  attribute  criteria,  a  key  part  of  any  specification  requirement  is  that  it  be  stated  in 
such  a  way  that  it  can  be  verified  (i.e.,  testable).  The  Chief  Developmental  Tester  should  be  involved  in 
specification  development  and  ideally  review  system  stated  requirements  from  this  perspective.  In  this 
regard,  paragraph  5.8  of  Military  Standard  MIL-STD-961E  (Reference  (ab)),  establishes  the  following 
criteria  for  stating  requirements: 

•  Each  requirement  shall  be  stated  in  such  a  way  that  an  objective  verification  can  be  defined  for 
it. 

•  Each  requirement  shall  be  cross-referenced  to  the  associated  verification. 

•  Only  requirements  that  are  necessary,  measurable,  achievable,  and  verifiable  shall  be  included. 

•  Requirements  shall  be  worded  to  provide  a  definitive  basis  for  acceptance  or  rejection. 

•  Requirements  shall  be  described  in  a  manner  to  encourage  competition. 

•  Requirements  shall  be  worded  such  that  each  paragraph  only  addresses  one  requirement  or 
topic. 

3.8  Independent  T&E  Agencies 

The  PMO  Chief  Developmental  Tester  does  not  have  direct  control  over  Government-owned  test 
resources,  test  facilities,  test  ranges,  test  range  personnel,  etc.  Therefore,  the  Chief  Developmental 
Tester  must  depend  on  those  DT  or  OT  organizations  controlling  them  and  stay  involved  with  the  test 
agency  activities  to  maximize  opportunities  for  integrated  testing. 

The  amount  of  involvement  depends  on  the  item  being  tested;  its  complexity,  cost,  and  characteristics; 
the  length  of  time  for  testing;  the  amount  of  test  funds;  etc.  Usually,  the  nuts  and  bolts  detailed  test 
plans  and  procedures  are  written  by  the  test  organizations  controlling  the  test  resources  with  input  and 
guidance  from  the  PMO  Chief  Developmental  Tester.  The  Chief  Developmental  Tester  is  responsible  for 
ensuring  that  the  tests  (DT&E)  are  performed  using  test  objectives  based  on  the  specifications  and  that 
the  requirements  of  timeliness,  accuracy,  and  minimal  costs  are  met  by  the  test  program  design.  During 
the  testing,  the  Chief  Developmental  Tester  monitors  test  results.  The  test  agencies  (DT/OT)  submit  a 
copy  of  their  report  to  the  program  office  at  the  end  of  testing,  usually  to  the  office  of  the  Chief 
Developmental  Tester.  The  Army  is  the  only  Service  to  have  a  designated  independent  evaluation 
agency,  which  provides  test  data  analysis  and  feedback  to  the  program  office. 

3.9  PMO  Involvement  In  OT&E  Activities 

In  the  Government  PMO,  there  should  be  a  group  responsible  for  integrating  T&E.  Besides  being 
responsible  for  DT&E  support  to  the  PM,  this  group  should  be  responsible  for  program  coordination 
with  the  OT&E  agency.  Program  T&E  metrics  for  success  and  OT&E  entrance  criteria  should  be 
developed  considering  lessons  learned  from  T&E  (see  Figure  3-2).  The  offices  of  the  systems  engineer  or 
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the  Chief  Developmental  Tester  may  be  designated  to  provide  this  support  to  the  PM.  In  some  Services, 
responsibilities  of  the  Chief  Developmental  Tester  include  coordination  of  test  resources  for  all  phases 
of  OT&E. 

3.9.1  Contractor  Responsibilities 


*  Understand  the  policies 

*  Organize  for  T&E 

*  Keep  system  requirements  documents  current 

*  Agonize  over  system  thresholds 

*  Work  closely  with  the  operational  test  director 

*  Don't  forget  about  operational  suitability 

*  Make  final  DT&E  a  rehearsal  for  lOT&E 

*  Prepare  interfacing  systems  for  your  lOT&E 

*  Manage  software  testing  closely 

*  Track  availability  of  test  resources  and  test  support 
personnel/facilities 

*  Identify  the  lack  of  any  test  resource  capability  to 
DASD(TT&E}  and  DOT&E 

Source:  Matt  Reynolds,  NAVSEA 

Figure  3-2  Some  T&E  "Lessons  Learned" 

The  Deputy  for  T&E  or  a  T&E  representative  ensures  that  certain  sections  of  the  REP  contain  sufficient 
allowance  for  T&E  support  by  contractors.  This  applies  whether  the  contract  is  for  a  developmental 
item,  a  production  item  (limited  production,  such  as  LRIP  or  FRP),  or  the  enhancement/upgrade  of 
portions  of  a  weapons  system.  Where  allowed  within  the  law,  contractor  support  for  OT&E  should  be 
considered  to  help  resolve  basic  issues  such  as  data  collection  requirements,  test  resources,  contractor 
test  support,  and  funding.  Chief  Developmental  Tester  or  a  T&E  representative  ensures  that  certain 
sections  of  the  RFP  contain  sufficient  allowance  for  T&E  support  by  contractors.  This  allowance  applies 
whether  the  contract  is  for  a  developmental  item,  a  production  item  (limited  production,  such  as  LRIP  or 
FRP),  or  the  enhancement/upgrade  of  portions  of  a  weapons  system.  Where  allowed  within  the  law. 
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contractor  support  for  OT&E  should  be  considered  to  help  resolve  basic  issues  such  as  data  collection 
requirements,  test  resources,  contractor  test  support,  and  funding. 

In  the  overall  portion  of  the  RFP,  Government  personnel,  especially  those  in  the  OTAs,  must  be 
guaranteed  access  to  the  contractor's  development  facilities,  particularly  during  the  DT&E  phase. 
Government  representatives  must  be  allowed  to  observe  selected  contractor  in-house  testing  and  have 
access  to  contractually  deliverable  test  data  and  reports. 

3.9.2  Technical  Data  Requirements 

The  contract  must  specify  what  data  the  contractor  will  supply  to  the  OTA.  Unlike  DT&E,  the  contractor 
will  not  be  creating  the  OT&E  plans,  procedures,  or  reports.  These  documents  are  the  responsibility  of 
the  OTA.  Section  2399d  of  Title  10,  U.S.C.  (Reference  (b)),  restricts  the  contractor  from  creating  the 
OT&E  plans,  procedures,  or  reports  or  participating  in  the  development  of  those  documents.  The  PMO 
Chief  Developmental  Tester  should  include  the  OTA  on  the  distribution  list  for  all  test  documents  that 
are  of  concern  during  the  DT&E  phase  of  testing  so  they  will  be  informed  of  test  item  progress  and 
previous  testing.  In  this  way,  the  OTA  will  be  informed  when  developing  its  own  test  plans  and 
procedures  for  OT&E.  In  fact,  the  OTA  should  provide  the  PMO  with  a  list  of  the  types  of  documents  the 
OTA  will  need,  and  an  OTA  representative  should  attend  the  CDRL  Review  Board  meeting.  The  Chief 
Developmental  Tester  should  coordinate  the  test  sections  of  this  data  list  with  the  OTA  and  indicate 
concerns  at  that  meeting.  All  contractor  test  reports  should  be  made  available  to  the  OTA.  In  return,  the 
Chief  Developmental  Tester  must  stay  informed  of  all  OTA  activities,  understand  the  test  procedures 
and  plans,  and  receive  the  test  reports.  Unlike  DT&E,  the  PMO  Chief  Developmental  Tester  will  not  have 
report  or  document  approval  authority  as  does  the  Chief  Developmental  Tester  over  contractor 
documentation.  The  Chief  Developmental  Tester  is  always  responsible  for  keeping  the  PM  informed  of 
OT&E  results. 

3.9.3  Test  Schedules 

Another  important  early  activity  that  the  Chief  Developmental  Tester  must  accomplish  is  to  coordinate 
the  OT&E  test  schedule.  If  the  contractor  is  required  to  provide  support  to  the  OT&E,  then  that  support 
must  be  included  in  the  SOW  and  RFP  and  placed  in  the  contract.  Sometimes,  the  Chief  Developmental 
Tester  can  formulate  a  draft  schedule  (based  on  previous  experience)  and  present  this  schedule  to  the 
operational  test  representative  at  the  initial  Test  Planning  Working  Group  (TPWG)  for  review,  or  the 
Chief  Developmental  Tester  can  contact  the  OTA  and  arrange  a  meeting  to  discuss  the  new  program.  In 
the  meeting,  time  requirements  envisioned  by  the  OTA  can  be  discussed.  Input  from  that  meeting  then 
goes  into  the  RFP  and  to  the  PM. 

The  test  schedule  must  allow  time  for  DT&E  and  OT&E  if  testing  is  not  combined  or  test  assets  are 
limited.  Before  setup  of  lOT&E,  the  certification  of  readiness  for  lOT&E  process  may  require  a  time  gap 
for  review  of  DT&E  test  results  and  refurbishment  or  corrections  of  deficiencies  discovered  during  DT&E, 
etc.  The  test  schedule  for  DT&E  should  not  be  so  success  oriented  that  the  lOT&E  test  schedule  is 
adversely  impacted,  not  allowing  enough  time  for  adequate  OT  or  the  reporting  of  lOT&E  results. 
Additionally,  the  overall  program  schedule  must  allow  for  sufficient  evaluation  time  following  an  OT 
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event  to  allow  the  OTA  to  make  a  well-informed  recommendation  for  the  coming  acquisition  program 
decision. 

3.9.4  Contractor  Support 

Sections  2399(d)  and  (e)  of  Reference  (b),  place  limits  on  contractor  involvement  in  lOT&E  of  ACAT  I  and 
II  programs.  A  long-standing  DoD  best  practice  has  been  to  apply  these  limitations  to  all  OT&E  programs 
regardless  of  ACAT.  The  Chief  Developmental  Tester  and  the  T&E  IPT  must  recognize  some  key 
distinctions  between  different  types  of  contractors  and  the  limitations  placed  on  each  type.  The  Chief 
Developmental  Tester  and  the  T&E  IPT  must  provide  inputs  to  the  SOW,  REP,  and  other  documents  with 
regard  to  contractor  support.  Some  contractor  involvement  is  permitted  in  OT&E. 

3. 9.4.1  System  Contractors 

According  to  section  2399(d)  of  Reference  (b),  operational  testers  must  strictly  avoid  situations  in  which 
system  contractors  could  reduce  the  credibility  of  OT  results  or  compromise  the  realistic 
accomplishment  of  OT  scenarios.  Section  2399(d)  of  Reference  (b)  permits  limited  system  contractor 
involvement  in  OT  if  the  operator  plans  for  the  contractor  to  be  involved  in  the  operation,  maintenance, 
and  support  of  the  system  when  deployed  in  combat.  If  not,  no  system  contractor  support  is  allowed 
during  lOT&E.  Any  deviations/waivers  must  be  approved  by  the  DOT&E. 

3. 9.4.2  System  Contractor  Support  to  OT 

System  contractors  may  be  beneficial  in  providing  logistic  support  and  training,  test  failure  analyses,  test 
data,  and  unique  software  and  instrumentation  support  that  could  increase  the  value  of  OT  data. 
Explanations  of  how  this  contractor  support  will  be  used  and  the  mitigation  of  possible  adverse  effects 
must  be  described  in  the  TEMP  and  developmental  and  operational  test  plans. 

3. 9.4.3  Support  Contractors 

According  to  section  2399(e)  of  Reference  (b),  support  contractors  may  not  be  involved  in  the 
establishment  of  criteria  for  data  collection,  performance  assessment,  or  evaluation  activities  for  OT. 
This  limitation  does  not  apply  to  a  support  contractor  that  has  participated  in  such  development, 
production,  or  testing  solely  in  test  or  test  support  on  behalf  of  the  Government. 

3.9.5  SOW 

One  of  the  most  important  documents  receiving  input  from  the  Chief  Developmental  Tester  is  the  SOW. 
The  Chief  Developmental  Tester  must  outline  all  required  or  anticipated  contractor  support  for  DT&E 
and  OT&E.  The  SOW  outlines  data  requirements,  contractor-conducted  or  supported  testing. 
Government  involvement  (access  to  contractor  data,  tests,  and  results),  OT  support,  and  any  other 
specific  test  requirements  the  contractor  will  be  tasked  to  perform  during  the  duration  of  the  contract. 
Additional  information  is  available  in  Reference  (y). 
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3.9.6  OT&E  Funding 

The  Chief  Developmental  Tester  provides  the  PM  with  estimates  of  PMO  test  program  costs  to  conduct 
OT&E.  This  funding  includes  contractor  and  Government  test  support  for  which  the  program  office 
directly  or  indirectly  will  be  responsible.  Because  Service  OTAs  fund  differently,  program  office  funding 
for  conducting  OT&E  varies.  The  Chief  Developmental  Tester  must  determine  these  costs  and  inform  the 
PM. 

3.9.7  T&E  Execution 

The  Chief  Developmental  Tester  is  responsible  for  managing  execution  of  the  T&E  strategy  within  the 
TEMP  throughout  the  test  program.  The  OTA  usually  is  tasked  to  complete  the  OT  portion  of  the  TEMP 
and  outline  its  proposed  test  program  through  all  phases  of  OT&E.  The  resources  required  for  OT&E 
should  also  be  entered  in  the  appropriate  section  of  the  TEMP.  It  is  important  to  keep  the  TEMP 
updated  regularly  so  that  test  organizations  involved  in  OT&E  understand  the  scope  of  their  test 
support.  Furthermore,  if  any  upgrades,  improvements,  or  enhancements  to  the  fielded  weapon  system 
occur,  the  TEMP  must  be  updated  or  a  new  one  created  to  outline  new  integrated  testing  and  OT 
requirements. 

3.9.8  PMO  Support  for  OT&E 

Even  though  OT  is  performed  by  an  independent  organization,  the  PM  plays  an  important  role  in  its 
planning,  reporting,  and  funding.  The  PM  must  coordinate  program  activities  with  the  test  community  in 
the  T&E  WIPT,  especially  the  OTAs.  The  OTA  ensures  that  testing  can  address  the  critical  issues.  The  PM 
should  provide  feedback  from  OT&E  activities  to  contractors. 

At  each  milestone  review,  the  PM  is  required  to  brief  the  decision  authority  on  the  testing  planned  and 
completed  on  the  program.  It  is  important  that  PMO  personnel  have  a  good  understanding  of  the  test 
program  and  that  they  work  with  the  OT  community  to  ensure  that  OT&E  is  well  planned  and  that 
adequate  resources  are  available.  The  PMO  should  involve  the  test  community  by  organizing  test 
coordinating  groups  at  program  initiation  and  by  establishing  channels  of  communication  between  the 
PMO  and  the  key  test  organizations.  The  PMO  can  often  avoid  misunderstandings  by  aggressively 
monitoring  the  system  testing  and  providing  up-to-date  information  to  key  personnel  in  OSD  and  the 
Services.  The  PMO  staff  should  keep  appropriate  members  of  the  test  community  well-informed 
concerning  system  problems  and  the  actions  taken  by  the  PMO  to  correct  them.  The  PMO  must  ensure 
that  contractor  and  Government  DT&E  supports  the  decision  to  certify  the  systems  readiness  for  lOT&E. 

3.9.9  Support  for  lOT&E 

DASD(DT&E)  conducts  AOTRs  for  all  MDAPs  and  programs  designated  as  Special  Interest  prior  to 
proceeding  to  lOT&E.  Procedurally,  DASD(DT&E)  produces  an  assessment  memorandum  prior  to  the 
lOT&Es  OTRR,  or  for  Space  Systems,  prior  to  consent  to  ship.  The  CAE  considers  the  results  of  the  DT&E 
assessment  in  order  to  determine  the  systems  materiel  readiness  for  lOT&E. 
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The  AOTR  is  a  DASD(DT&E)  conducted  analysis,  beginning  at  Milestone  B,  that  assesses  the  results  of 
developmental  testing  and  determines  readiness  to  proceed  to  lOT&E.  This  analysis  is  based  upon  a 
memorandum  signed  by  DASD(DT&E)  and  distributed  to  USD(AT&L),  DOT&E  ,  CAEs  and  Service  T&E 
Executives.  Each  AOTR  evaluates  system  performance  (against  key  performance  measures  (e.g.  KPP, 

KSA,  CTP,  etc.),  reliability,  interoperability,  and  information  assurance  as  well  as  the  risks  associated  with 
the  system's  ability  to  meet  its  operational  suitability,  effectiveness,  and  survivability  requirements.  The 
AOTR  bases  its  findings  and  recommendations  on  the  results  of  all  program  T&E  conducted  to  date, 
including  full-up  system  level  DT&E,  integrated  tests,  certifications,  and  prior  operational 
assessments(s). 

The  certification  of  readiness  for  the  lOT&E  process  provides  a  forum  for  a  final  review  of  test  results 
and  support  required  by  the  lOT&E.  The  Chief  Developmental  Tester  must  ensure  that  the  contract 
portions  adequately  cover  the  scope  of  testing  as  outlined  by  the  OTA.  The  program  office  may  want  to 
provide  an  observer  to  represent  the  Chief  Developmental  Tester  during  the  actual  testing.  The  key 
mission  of  Chief  Developmental  Tester  involvement  in  lOT&E  will  be  to  monitor  and  coordinate;  the 
Chief  Developmental  Tester  will  keep  the  PM  informed  of  progress  and  problems  that  arise  during 
testing  and  will  monitor  required  PMO  support  to  the  test  organization.  Sufficient  LRIP  items  plus  spare 
parts,  training  materials,  and  other  logistics  products  must  be  provided  to  the  OTA  to  enable  it  to  run  a 
complete  and  adequate  OT&E  program.  The  DOT&E  determines  the  number  of  LRIP  items  for  lOT&E  on 
oversight  programs,  and  the  OTA  makes  this  determination  for  all  others.  For  problems  requiring 
program  office  action,  the  Chief  Developmental  Tester  will  be  the  point  of  contact. 

The  Chief  Developmental  Tester  will  be  concerned  with  lOT&E  of  the  LRIP  units  after  a  limited  number 
are  produced.  The  lOT&E  must  be  closely  monitored  so  that  an  FRP  decision  can  be  made.  As  in  the 
operational  assessments,  the  Chief  Developmental  Tester  will  be  monitoring  test  procedures  and  results 
and  keeping  the  PM  informed.  If  the  item  does  not  succeed  during  lOT&E,  a  new  process  of  DT&E  or  a 
modification  may  result;  and  the  Chief  Developmental  Tester  will  be  involved  (as  in  any  new  programs 
inception).  If  the  item  passes  lOT&E  and  is  produced  at  full  rate,  the  Chief  Developmental  Tester  will  be 
responsible  for  ensuring  that  testing  of  those  production  items  is  adequate  to  ensure  that  the  end  items 
physically  and  functionally  resemble  the  developmental  items. 

3.9.10  FOT&E  and  Modifications,  Upgrades,  Enhancements,  or  Additions 

During  FOT&E,  the  Chief  Developmental  Tester  monitors  the  testing.  The  level  of  contractor 
involvement,  if  any,  is  usually  negotiated  with  the  OTA.  The  Chief  Developmental  Tester  should  receive 
any  reports  generated  by  the  operational  testers  during  this  time.  Any  deficiencies  noted  during  FOT&E 
should  be  evaluated  by  the  PMO,  which  may  decide  to  incorporate  upgrades,  enhancements,  or 
additions  to  the  current  system  or  in  future  increments.  If  the  PM  and  the  engineering  section  of  the 
program  office  design  or  develop  modifications  that  are  incorporated  into  the  weapon  system  design, 
additional  FOT&E  may  be  required. 

Once  a  weapon  system  is  fielded,  portions  of  that  system  may  become  obsolete,  ineffective,  or  deficient 
and  may  need  to  be  replaced,  upgraded,  or  enhanced  to  ensure  that  the  weapon  system  meets  current 
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and  future  requirements.  The  Chief  Developmental  Tester  plays  a  vital  role  in  this  process.  Modifications 
to  existing  weapon  systems  may  be  managed  as  an  entire  newly  acquired  weapon  system.  However, 
since  these  are  changes  to  existing  systems,  the  Chief  Developmental  Tester  is  responsible  for 
determining  whether  these  enhancements  degrade  the  existing  system,  whether  they  are  compatible 
with  its  interfaces  and  functions,  and  whether  nondevelopmental  items  (NDIs)  require  retest,  or  the 
entire  weapon  system  needs  reverification.  The  Chief  Developmental  Tester  must  plan  the  test 
programs  funding,  schedule,  test  program,  and  contract  provisions  with  these  items  in  mind.  A  new 
TEMP  may  have  to  be  generated  or  the  original  weapon  system  TEMP  modified  and  recoordinated  with 
the  test  organizations.  The  design  of  the  DT&E  and  FOT&E  program  usually  requires  coordination  with 
the  engineering,  contracting,  and  program  management  IPTs  of  the  program  office  as  well  as  the 
supporting  DT  and  OT  commands,  ranges,  and  agencies. 

3.9.11  Test  Resources 

During  all  phases  of  T&E,  the  Chief  Developmental  Tester  must  coordinate  with  the  participating 
Services  and  the  operational  testers  to  ensure  that  they  have  the  test  articles  needed  to  accomplish 
their  mission.  Test  resources  will  be  provided  by  the  contractor  or  the  Government.  The  contractor 
resources  must  be  covered  in  the  contract,  whether  in  the  development  contract  or  the  production 
contract.  Government  test  resources  needed  are  estimated  by  the  operational  testers.  They  usually 
coordinate  the  test  ranges,  test  support,  and  the  user  personnel  for  testing.  The  PM  programs  funding 
for  support  of  OT.  Funding  for  OT&E  is  identified  in  the  TEMP  and  funded  in  the  PMOs  budget.  Each 
Service's  policy  specifies  how  the  OTAs  develop  and  manage  their  own  budgets  for  OT.  See  Chapter  13 
for  more  information  on  test  resources. 

3.10  Summary 

Staffing  requirements  in  the  PMO  vary  with  the  program  phase  and  the  T&E  workload.  The  chief 
developmental  tester  must  be  a  designated  KLP  for  MDAPs  and  MAIS  programs  and  ideally  should  be  a 
separate  staff  element  within  the  program  office.  T&E  expertise  is  essential  in  the  early  planning  stages 
but  can  be  provided  through  matrix  support. 

The  PMO  should  be  proactive  in  its  relations  with  the  Service  OTA.  Many  opportunities  exist  for  valuable 
mutual  support,  collaboration,  and  information  sharing.  Early  OTA  input  to  design  considerations  and 
requirements  clarification  can  reduce  surprises  downstream  and  can  facilitate  integrated  testing  efforts. 
OT  is  an  essential  component  of  the  system  development  and  decision-making  process.  OT  should  be 
used  to  facilitate  system  development  and  not  become  part  of  an  adversarial  process. 
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Test  &  Evaluation  Management  Guide 
Chapter  4  -  Test  -  Related  Documentation 

4.1  Introduction 

During  the  course  of  a  defense  acquisition  program,  many  documents  are  developed  that  have 
significance  for  those  responsible  for  testing  and  evaluating  the  system.  This  chapter  provides 
background  on  some  of  these  documents.  Because  the  information  provided  in  this  chapter  may  change 
over  time;  consequently,  the  reader  should  see  Reference  (j)  as  well  as  any  Service-unique  publications 
for  more  current  information. 

A  pictorial  roadmap  of  key  activities  in  the  systems  acquisition  processes-the  Integrated  Defense 
Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System  chart-is  available  at 
https://ilc.dau.mil/html/ILC  Main.htm  .  The  chart  is  a  training  aid  for  DAU  courses  and  illustrates  the 
interaction  of  the  three  key  processes  that  must  work  in  concert  to  deliver  the  capabilities  required  by 
the  Warfighters:  the  requirements  process  (JCIDS  process);  the  acquisition  process  (Defense  Acquisition 
System  process);  and  the  program  and  budget  development  process  (PPBE  process). 

As  Figure  4-1  shows,  test-related  documentation  spans  a  broad  range  of  materials.  It  includes 
requirements  documentation  such  as  the  CDD  and  CPD;  program  decision  documentation  such  as  the 
ADM  with  exit  criteria;  program  management  documentation  such  as  the  Acquisition  Strategy,  APB,  and 
waivers  (i.e.,  LFT&E  waiver  from  FUSE  testing);  and  test  program  documentation  such  as  LFT&E  Strategy, 
the  System  Evaluation  Plan  (SEP),  and  the  TEMP. 


Figure  4-1  Examples  of  T&E-Related  Program  Plans  and  Documentation 
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Of  importance  to  PMs  and  to  T&E  managers  are  additional  test-unique  program  documents  such  as 
specific  test  designs,  test  plans,  Outline  Test  Plans  (OTP)/Test  Program  Outlines,  evaluation  plans,  and 
test  reports.  Prior  to  lOT&E,  the  DASD(DT&E)  provides  an  AOTR  showing  risks  to  lOT&E  by  predicting 
unfavorable  results  and  eliminating  the  added  time  and  expense  needed  when  having  to  retest  failures. 
This  chapter  concludes  with  a  description  of  the  end-of-test  phase  and  BLRIP  reports,  and  two  special- 
purpose  T&E  status  reports  that  are  used  to  support  the  milestone  decision  process. 

4.2  Requirements  Documentation 

4.2.1  Continuing  Mission  Capability  Analyses 

The  Joint  Chiefs  of  Staff,  via  the  Joint  Requirements  Oversight  Council  (JROC),  are  required  to  conduct 
continuing  mission  analyses  of  their  assigned  areas  of  responsibility  in  accordance  with  CJCS  Instruction 
3170.01H  (Reference  (ad)).  These  functional  analyses  (area,  needs,  and  solutions)  may  result  in 
recommendations  to  initiate  new  acquisition  programs  to  reduce  or  eliminate  operational  deficiencies. 

If  a  need  cannot  be  met  through  changes  in  doctrine,  organization,  training,  leadership  and  education, 
personnel,  and  facilities,  and  a  materiel  solution  is  required,  the  needed  capability  is  described  first  in  an 
ICD,  and  then  in  the  CDD  (See  Figure  4-2).  When  the  cost  of  a  proposed  acquisition  program  is  estimated 
to  meet  certain  cost  limits  specified  in  Reference  (d),  the  proposed  program  is  considered  an  MDAP  and 
requires  JROC  approval  (JROC  Interest).  Lesser  programs  for  Service  action  may  be  designated  as  Joint 
Capabilities  Board  Interest,  Joint  Integration,  Joint  Information,  or  Independent  programs.  The  ICD  is 
completed  at  the  beginning  of  the  analyses  for  a  mission  area  (before  the  MDD  and  the  MSA  effort).  The 
CDD  at  MS  B  and  CPD  at  MS  C  are  program  and/or  increment-specific  and  reviewed  to  evaluate 
necessary  system  modifications  periodically. 

4.2.2  ICD 

The  ICD  makes  the  case  to  establish  the  need  for  a  materiel  approach  to  resolve  a  specific  capability  gap. 
The  ICD  supports  the  AoA,  development  of  the  TDS,  and  various  milestone  decisions.  The  ICD  defines 
the  capability  gap  in  terms  of  the  functional  area(s),  relevant  range  of  military  operations,  time, 
obstacles  to  overcome,  and  key  attributes  with  appropriate  MOEs.  Once  approved,  the  ICD  is  not 
normally  updated. 

4.2.3  CDD 

The  CDD  outlines  an  affordable  increment  of  militarily  useful  and  supportable  operational  capability  that 
can  be  effectively  developed,  produced  or  acquired,  deployed,  and  sustained.  Each  increment  of 
capability  will  have  its  own  set  of  attributes  and  associated  performance  values  with  thresholds  and 
objectives  established  by  the  sponsor  (Service)  with  input  from  the  user.  The  CDD  performance 
attributes  apply  to  only  one  increment  and  include  KPPs  and  other  parameters  that  will  scope 
performance  and  provide  the  trade  space  for  development  effort  (See  Reference  (ad) ). 
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4.2.4  CPD 


The  CPD  addresses  the  production  attributes  and  quantities  specific  to  a  single  increment  and  is  finalized 
by  the  sponsor  after  the  PCDRA,  when  projected  capabilities  of  the  increment  in  development  have 
been  specified  with  sufficient  accuracy  to  begin  production.  Design  trades  during  EMD  should  have  led 
to  the  final  production  design  needed  for  MS  C.  lOT&E  will  normally  test  to  the  values  in  the  CPD.  The 
threshold  and  objective  values  from  the  CDD  are  therefore  superseded  by  the  specific  production  values 
detailed  in  the  CPD.  Reduction  in  any  KPP  threshold  value  will  require  reassessment  of  the  military  utility 
of  the  reduced  capability.  The  CPD  will  always  reference  the  originating  ICD. 

4.2.5  STAR 


A  System  Threat  Assessment  Report  (STAR)  is  prepared,  starting  at  MS  B,  by  the  DoD  Component 
Intelligence  Command  or  Agency  for  ACAT  ID  programs,  and  is  validated  by  the  Defense  Intelligence 
Agency  in  accordance  with  DoDD  5105.21  (Reference  (ae)).  The  STAR  is  a  concise  description  of  the 
projected  future  operational  threat  environment,  the  system-specific  threat,  the  reactive  threat  that 
could  affect  program  decisions,  and  when  appropriate,  the  results  of  interactive  analysis  obtained  when 
evaluating  the  program  against  the  threat.  Threat  projections  start  at  IOC  and  extend  over  the  following 
10  years.  The  STAR  provides  the  basis  for  the  test  design  of  threat  scenarios  and  the  acquisition  of 
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appropriate  threat  targets,  equipment,  or  surrogates.  It  provides  threat  data  for  DT&E  and  OT&E. 
Vulnerability  and  lethality  analyses  during  LET  of  ACAT  I  and  II  systems  are  contingent  on  valid  threat 
descriptions.  A  summary  of  the  STA  should  be  included  in  Part  I  of  the  TEMP. 

4.2.6  Importance  of  Current  Requirements  Documents 

Milestone  decisions  and  OSD  test  plan  approval  are  highly  dependent  on  having  the  most  current 
approved  JCIDS  documents.  Out-of-date  JCIDS  documents  are  unacceptable  and  put  the  programs  T&E 
strategy-and  ultimately  the  program  itself-at  risk.  Delays  in  approval  run  the  risk  of  having  to  reschedule 
milestone  decisions  or  non-approval  of  critical  test  plans  by  DOT&E.  The  OTA  will  insist  on  testing  to  the 
latest  available  threat  information,  and  if  that  is  not  reflected  in  the  latest  threat  and  JCIDS  documents, 
conflicts  can  occur  that  may  derail  the  program. 

4.2.7  Testability  of  Requirements 

The  Chief  Developmental  Tester  and  SMEs  from  the  T&E  WIPT  must  participate  in  early  requirements 
development  forums  as  soon  as  a  new  program  is  considered.  The  objective  is  not  to  write  requirements 
for  users  but  to  help  the  users  articulate  better  requirements  as  clearly  as  possible  and  ensure  that  the 
requirements  can  be  tested  (i.e.,  the  requirements  are  testable).  DT&E  personnel  can  provide  advice 
about  the  technical  feasibility  of  proposed  new  requirements,  making  the  system  more  affordable  and 
more  technologically  achievable.  Operational  testers  can  help  ensure  that  the  system  (1)  can  meet 
mission  requirements,  (2)  clearly  conveys  the  users'  needs,  and  (3)  can  be  determined  to  be 
operationally  effective  and  suitable  with  a  high  degree  of  confidence. 

4.3  Program  Decision  Documentation 

4.3.1  ADM 

USD(AT&L)  decisions  at  major  defense  ACAT  ID  milestones  are  recorded  in  a  document  known  as  an 
ADM.  The  ADM  documents  a  USD(AT&L)  decision  on  program  progress  and  on  the  APB  at  milestones 
and  decision  reviews.  In  conjunction  with  an  ADM,  and  its  included  exit  criteria  for  the  next  phase,  the 
APB  is  a  primary  program  guidance  document  providing  KPP  thresholds/objectives  for  systems  cost, 
schedule,  and  performance  and  phase  success  criteria. 

4.3.2  AoA 

The  AoA  is  an  important  element  of  the  defense  acquisition  process.  It  is  an  analytical  comparison  of  the 
operational  effectiveness,  suitability,  and  life-cycle  cost  (or  total  ownership  cost,  if  applicable)  of 
alternatives  that  satisfy  established  capability  needs.  Initially,  after  the  MDD,  the  AoA  is  initiated  to 
examine  potential  materiel  solutions  with  the  goal  of  identifying  the  most  promising  option,  thereby 
guiding  the  MSA  phase.  For  MDAPs  at  MS  A,  the  MDA  must  certify  to  Congress  that  the  Department  has 
completed  an  AoA  consistent  with  study  guidance  developed  by  the  OSD  Director  of  Cost  Assessment 
and  Program  Evaluation  (DCAPE).  For  MDAPs  at  MS  B,  the  MDA  must  certify  in  writing  to  Congress  that 
the  Department  has  completed  an  AoA  with  respect  to  the  program.  An  AoA  is  normally  not  required  at 
MS  C  unless  significant  changes  to  threats,  cost,  or  technology  have  occurred,  or  the  analysis  is 
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otherwise  deemed  necessary  by  the  MDA.  The  DCAPE  develops  and  approves  study  guidance  for  the 
AoA.  The  guidance  is  developed  with  input  from  other  DoD  officials,  and  DCAPE  provides  the  approved 
guidance  at  the  MDD  review.  Following  a  successful  MDD  review,  the  MDA  directs  initiation  of  the  AoA 
in  the  ADM  and  includes  the  AoA  study  guidance  as  an  attachment  to  the  ADM.  Following  receipt  of  the 
AoA  study  guidance,  the  lead  DoD  Component  prepares  an  AoA  study  plan  that  describes  the  intended 
methodology  for  the  management  and  execution  of  the  AoA.  The  study  plan  is  coordinated  with  the 
MDA  and  approved  by  the  DCAPE  prior  to  the  start  of  the  AoA.  The  AoA  aids  decision  makers  by 
examining  the  relative  advantages  and  disadvantages  of  program  alternatives,  shows  the  sensitivity  of 
each  alternative  to  possible  changes  in  key  assumptions,  and  provides  the  rationale  for  each  option. 
There  should  be  a  clear  linkage  between  the  AoA,  system  requirements,  and  system  evaluation  criteria. 
One  driving  factor  for  ensuring  that  this  linkage  exists  is  the  decision  maker's  acceptance  of  M&S 
projections  or  analytic  studies  for  system  performance  in  the  future  without  actual  test  data  that 
validate  AoA  results.  In  addition  to  a  final  briefing,  the  AoA  process  and  results  are  usually  documented 
in  a  final  written  report. 

Reference  (d)  requires  an  AoA  for  MAIS  programs  at  milestone  decisions.  The  AoA  for  MAIS  programs 
should  include  a  discussion  of  whether  the  proposed  program  (1)  supports  a  core/priority  mission  or 
function  performed  by  the  DoD  Component,  (2)  needs  to  be  undertaken  because  no  alternative  private 
sector  or  governmental  source  can  better  support  the  function,  and  (3)  supports  improved  work 
processes  that  have  been  simplified  or  otherwise  redesigned  to  reduce  costs,  improve  effectiveness,  and 
make  maximum  use  of  commercial  off-the-shelf  (COTS)  technology.  Most  likely,  the  effectiveness 
analysis  in  a  MAIS  program  AoA  will  not  involve  scenario-based  analysis  as  is  common  for  the  weapon 
system  AoAs. 

4.4  Program  Management  Documentation 
4.4.1  Acquisition  Strategy 

An  event-driven  acquisition  strategy  must  be  formulated  at  the  start  of  a  development  program.  Event- 
driven  acquisition  strategy  explicitly  links  program  decisions  to  demonstrated  accomplishments  in 
development,  testing,  and  initial  production.  The  strategy  constitutes  a  broad  set  of  concepts  that 
provide  direction  and  control  for  the  overall  development  and  production  effort.  The  level  of  detail 
reflected  in  the  acquisition  strategy  can  be  expected  to  increase  as  a  program  matures.  The  acquisition 
strategy  serves  as  a  tailored  conceptual  basis  for  formulating  other  program  functional  plans  such  as  the 
TEMP. 

It  is  important  that  T&E  interests  be  represented  as  the  acquisition  strategy  is  formulated  because  the 
acquisition  strategy  should: 

•  Provide  an  overview  of  what  requires  evaluation  for  decision  points  and  the  planned  T&E  to 
ensure  that  all  decisions  are  supported  with  adequate  T&E  results. 

•  Discuss  plans  for  providing  adequate  quantities  of  test  hardware. 

•  Describe  test  activities  and  relationships  to  integrate  testing  across  DT&E  and  OT&E,  and  the 
level  of  concurrence  needed  to  implement  them. 
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4.4.2  APB 

Reference  (d)  requires  each  PM  to  propose  and  document  program  goals  prior  to  and  for  approval  at 
program  initiation  for  all  ACAT  programs.  The  APB  is  an  important  document  for  program  management 
and  should  reflect  the  approved  program  being  executed.  The  APB  will  initially  be  developed  before  MS 
B  or  program  initiation  and  will  be  revised  for  each  subsequent  milestone.  Baseline  parameters 
represent  the  cost,  schedule,  and  performance  objectives  and  thresholds  for  the  system  in  a  production 
configuration.  Each  baseline  influences  the  T&E  activities  in  the  succeeding  phases.  Guidance  on  the 
formulation  of  baselines  is  found  in  Chapter  9  of  Reference  (j).  Performance  demonstrated  during  T&E 
of  production  systems  must  meet  or  exceed  the  thresholds.  The  thresholds  establish  deviation  limits 
(actual  or  anticipated  breach  triggers  reports)  for  KPPs  beyond  which  the  PM  may  not  trade  off  cost, 
schedule,  or  performance  without  authorization  by  the  MDA.  Baseline  and  test  documentation  must 
reflect  the  same  expectations  for  system  performance.  The  total  number  of  KPPs  shall  be  the  minimum 
number  needed  to  characterize  the  major  drivers  of  operational  effectiveness  and  suitability,  schedule, 
technical  progress,  and  cost  for  a  production  system  intended  for  deployment.  The  performance 
parameters  may  not  completely  define  operational  effectiveness  or  suitability. 

4.4.3  Acquisition  Logistics  Planning  Documentation 

All  logistics  planning  must  be  derived  from  user-developed  JCIDS  operational  requirement  documents 
and  further  described  and  expanded  using  operating  and  enabling  concepts.  These  requirements  and 
concepts  are  expressed  in  TEMPs  and  test  plans  as  KPPs,  critical  operational  issues  (COIs),  measures  of 
suitability  (MOSs),  MOPs,  and  operational  mission  scenarios.  A  suitability  KPP  supported  by  MOEs, 

MOSs,  and  MOPs  is  mandatory. 

Supportability  analyses  are  a  composite  of  all  support  considerations  necessary  to  ensure  the  effective 
and  economical  support  of  a  system  at  all  levels  of  maintenance  for  its  programmed  life  cycle.  Support 
concepts  describe  the  overall  logistics  support  program  and  include  logistics  requirements,  tasks,  and 
milestones  for  the  current  and  succeeding  phases  of  the  program.  The  analyses  serve  as  the  source 
document  for  logistics  support  testing  requirements. 

MIL-HDBK-502  (Reference  (af))  documents  guidelines  for  logistics  support  analyses  (LSAs)  and  provides 
information  on  how  T&E  programs  should  be  planned  to  serve  the  following  three  logistics 
supportability  objectives: 

•  Provide  measured  data  for  input  into  system-level  estimates  of  readiness,  operational  costs,  and 
logistics  support  resource  requirements. 

•  Expose  supportability  problems  so  they  can  be  corrected  prior  to  deployment. 

•  Demonstrate  contractor  compliance  with  quantitative  supportability-related  design 
requirements. 

Development  of  an  effective  T&E  program  requires  close  coordination  of  efforts  among  all  SE  disciplines, 
especially  those  involved  in  LSAs.  Areas  of  particular  interest  include  RAM,  HSI,  environment,  safety  and 
occupational  health  (ESOH),  and  post-deployment  evaluations.  The  support  analyses  should  be  drafted 
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shortly  before  program  start  to  provide  a  skeletal  framework  for  LSA,  to  identify  initial  logistics  testing 
requirements  that  can  be  used  as  input  to  the  TEMP,  and  to  provide  test  feedback  to  support  ILS 
development. 

4.4.4  FUSL  LFT  and  Waiver  Process 

The  term  full-up,  system-level  live-fire  testing  is  the  testing  that  fully  satisfies  the  statutory  requirement 
for  realistic  survivability  testing  or  realistic  lethality  testing  as  defined  in  section  2366  of  Reference  (b). 
The  criteria  for  FUSL  LFT  differ  somewhat  depending  on  whether  the  testing  is  for  survivability  or 
lethality.  FUSL  testing  consists  of  the  following: 

Vulnerability  testing  conducted,  using  munitions  likely  to  be  encountered  in  combat,  on  a  complete 
system  loaded  or  equipped  with  all  the  dangerous  materials  that  normally  would  be  on  board  in  combat 
(including  flammables  and  explosives),  and  with  all  critical  subsystems  operating  that  could  make  a 
difference  in  determining  the  test  outcome;  or 

Lethality  testing  of  a  production-representative  munition  or  missile,  for  which  the  target  is 
representative  of  the  class  of  systems  that  includes  the  threat,  and  the  target  and  test  conditions  are 
sufficiently  realistic  to  demonstrate  the  lethal  effects  the  weapon  is  designed  to  produce. 

Section  2366  of  Reference  (b)  requires  an  LFT&E  program  to  include  FUSL  testing  unless  a  waiver  is 
granted  in  accordance  with  procedures  defined  by  the  statute.  A  waiver  package  must  be  sent  to  the 
congressional  defense  committees  prior  to  MS  B;  or  in  the  case  of  a  system  or  program  initiated  at  MS  B, 
as  soon  as  practicable  after  MS  B;  or  if  initiated  at  MS  C,  as  soon  as  practicable  after  MS  C.  Typically,  this 
request  for  a  waiver  should  occur  at  the  time  of  TEMP  approval. 

The  waiver  package  includes  certification  by  the  USD(AT&L)  or  the  DoD  CAE  that  FUSL  testing  would  be 
unreasonably  expensive  and  impractical.  It  also  includes  a  DOT&E-approved  alternative  plan  for 
conducting  LFT&E  in  the  absence  of  FUSL  testing.  Typically,  the  alternative  plan  is  similar  or  identical  to 
the  LFT&E  Strategy  contained  in  the  TEMP.  This  alternative  plan  should  include  LFT&E  of  components, 
subassemblies,  or  subsystems  and,  as  appropriate,  additional  design  analyses,  M&S,  and  combat  data 
analyses. 

Programs  that  have  received  a  waiver  from  FUSL  testing  are  conducted  as  LFT&E  programs  (with 
exception  of  the  statutory  requirement  for  FUSL  testing).  In  particular,  the  TEMP  contains  an  LFT&E 
Strategy  approved  by  DOT&E,  and  DOT&E,  as  delegated  by  the  SecDef,  submits  an  independent 
assessment  report  on  the  completed  LFT&E  to  the  congressional  committees  as  required  by  section 
2366  of  Reference  (b).  LFT&E  is  discussed  in  Chapter  16. 

4.4.5  System  Specification 

The  system  specification  is  a  key  technical  document  that  describes  both  the  technical  performance 
requirements  and  the  verification  of  these  requirements  for  the  system.  The  system  includes  items, 
software,  processes,  materials,  and  services.  Section  3  (Requirements)  of  the  specification  describes  the 
required  performance  parameters  and  Section  4  (Verification)  identifies  the  procedures  (analysis. 
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demonstration,  examination,  testing)  used  to  verify  these  parameters.  Other  specifications  generated 
over  the  developmental  phases  of  the  program  (with  increasing  levels  of  detail)  include  item 
performance  specifications,  item  detail  specifications,  process  specifications,  and  material  specifications. 
Reference  (ab)  provides  further  details. 

4.4.6  Work  Breakdown  Structure  (WBS) 

A  program  WBS  shall  be  established  that  provides  a  framework  for  program  and  technical  planning,  cost 
estimating,  resource  allocations,  performance  measurements,  and  status  reporting.  Program  offices 
shall  tailor  a  program  WBS  for  each  program  using  the  guidance  in  MIL-STD-881C  (Reference  (ag))  Level 
2  of  the  WBS  hierarchical  structure  typically  addresses  system-level  T&E  with  sublevels  for  DT&E  and 
OT&E.  Additionally,  each  configuration  item  (Cl)  WBS  should  include  details  of  relevant  integration  and 
test  requirements. 

4.4.7  SOW 

The  SOW  is  that  portion  of  a  contract  that  establishes  and  defines  all  non-specification  requirements  for 
contractor's  efforts  either  directly  or  with  the  use  of  specific  cited  documents.  The  SOW  details  the  work 
the  contractor  will  perform  and,  when  necessary,  specifies  how  the  work  is  to  be  performed.  Additional 
information  is  available  in  Reference  (y) . 

4.5  Test  Program  Documentation 

4.5.1  TES/TEMP 

The  TES  describes  the  concept  for  T&E  throughout  the  program  life  cycle  starting  with  Technology 
Development  and  continuing  through  EMD  into  Production  and  Deployment.  The  TES  evolves  into  the 
TEMP  at  Milestone  B.  Development  of  a  TES  requires  early  involvement  of  testers,  evaluators,  user 
representatives,  etc.,  as  a  program  conducts  pre-system  activities.  These  personnel  provide  the 
necessary  technical,  operational,  and  programmatic  expertise  to  ensure  the  development  of  a 
comprehensive  strategy.  The  TES  approval  process  is  explained  in  section  9. 5. 4.3  of  the  DAG. 

The  TEMP  is  the  basic  planning  document  for  DoD  system  acquisition  T&E.  It  is  the  PM/PMO  contract 
with  the  programs  stakeholders  on  how  T&E  will  be  conducted  and  is  prepared  by  the  PMO  with  the  OT 
information  provided  by  the  Service  OTA.  The  TEMP  is  used  by  OSD  and  the  Services  for  planning, 
reviewing,  and  approving  T&E  programs  and  provides  the  basis  and  authority  for  all  other  detailed  T&E 
planning  documents.  The  evaluation  framework  in  the  TEMP  identifies  the  key  requirements  (KPPs  and 
KSAs),  COIs  (or  COICs  for  Army  TEMPS),  critical  technical  parameters  (CTPs),  MOEs,  MOSs,  major  T&E 
resources  required,  and  decision  supported.  The  TEMP  format  is  outlined  in  detail  in  section  9.5.5  of 
Reference  (j). 

4.5.1.1  LFT&E  Strategy 

The  objective  of  LFT&E  is  to  provide  a  timely  and  reasonable  assessment  of  the  vulnerability/lethality  of 
a  system  as  it  progresses  through  its  development  and  prior  to  FRP.  The  LFT&E  Strategy  for  a  given 
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system  should  be  structured  and  scheduled  so  that  any  design  changes  resulting  from  the  testing  and 
analysis,  described  in  the  LFT&E  Strategy,  may  be  incorporated  before  proceeding  beyond  LRIP.  LFT&E 
testing  is  discussed  in  Chapter  16. 

The  DOT&E  approves  the  adequacy  of  the  LFT&E  Strategy  before  the  program  begins  LFT&E.  The 
program  should  be  driven  by  LFT&E  issues  identified  in  the  strategy  and  be  fully  integrated  with  planned 
DT&E  and  OT&E.  LFT&E  typically  includes  testing  at  the  component,  subassembly,  and  subsystem  level 
and  may  also  draw  upon  design  analyses,  M&S,  combat  data,  and  related  sources  such  as  analyses  of 
safety  and  mishap  data.  This  approach  is  standard  practice,  regardless  of  whether  the  LFT&E  program 
culminates  with  FUSE  testing,  or  a  waiver  from  FUSE  testing  is  obtained. 

4.5. 1.2  Alternative  LFT&E  Strategy 

Programs  that  have  received  a  waiver  from  FUSE  testing  are  conducted  as  LFT&E  programs  (with  the 
exception  of  the  statutory  requirement  for  FUSE  testing).  The  waiver  package  includes  a  DOT&E- 
approved  alternative  plan  for  conducting  LFT&E  in  the  absence  of  FUSE  testing.  Typically,  the  alternative 
plan  is  similar  or  identical  to  the  LFT&E  Strategy  contained  in  the  TEMP. 

4.5.2  System  Evaluation  Plan 

Reference  (d)  requires  PMs  to  prepare  a  system  evaluation  plan  for  each  milestone  review,  beginning 
with  MS  A.  At  MS  A,  the  system  evaluation  plan  supports  the  TDS;  at  MS  B  or  later,  the  system 
evaluation  plan  supports  the  Acquisition  Strategy.  The  system  evaluation  plan  describes  the  programs 
overall  technical  approach,  including  key  technical  risks,  processes,  resources,  metrics,  and  applicable 
performance  incentives.  It  also  details  the  timing,  conduct,  and  success  criteria  of  technical  reviews.  The 
TES  should  be  consistent  with  and  complementary  to  the  system  evaluation  plan  and  Acquisition 
Strategy.  The  T&E  team  should  work  closely  with  the  PM  and  the  system  design  team  to  facilitate  this 
process. 

4.5.3  Test  Design 

Test  designers  need  to  ensure  that  the  test  is  constructed  to  provide  useful  information  in  all 
areas/aspects  that  will  lead  to  an  assessment  of  the  system  performance.  A  well-designed  experiment 
can  lead  to  reduced  development  lead  time  with  fewer  tests  required,  provide  greater  insight  to  system 
performance,  and  ultimately  lead  to  fielding  better,  more  reliable  systems.  The  use  of  ST AT  in  T&E  will 
generate  T&E  efficiencies;  provide  rigorous,  defensible  T&E  strategies  and  results;  and  improve  the  level 
of  knowledge  for  the  DT  planning,  execution,  and  analysis  process.  When  used  in  the  proper  context, 

ST AT  in  T&E  will  enable  the  PMs  to  make  better-informed  decisions  based  on  acceptable  risk  thresholds. 
For  example,  a  complicated,  even  ingenious,  test  that  does  not  provide  the  information  required  by  the 
decision  makers  is,  in  many  respects,  a  failed  endeavor.  Therefore,  part  of  the  process  of  developing  a 
test  concept  or  test  design  (the  distinction  between  these  vary  from  organization  to  organization) 
should  be  to  consider  whether  the  test  will  provide  the  information  required  by  the  decision  makers.  In 
other  words:  Are  we  testing  the  right  things  in  the  right  way?  Are  our  evaluations  meaningful? 
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The  test  design  should  be  statistical  and  analytical  in  nature  and  should  perform  the  following  functions: 

•  Structure  and  organize  the  approach  to  testing  in  terms  of  specific  test  objectives. 

•  Identify  key  MOEs  and  MOPs. 

•  Identify  the  required  data  and  illustrate  how  the  data  will  be  gathered,  stored,  analyzed,  and 
used  to  evaluate  MOEs. 

•  Indicate  what  part  M&S  will  play  in  meeting  test  objectives. 

•  Identify  the  number  and  type  of  test  events  and  required  resources. 

The  test  design  may  serve  as  a  foundation  for  the  more  detailed  test  plan  and  specifies  the  test 
objectives,  events,  instrumentation,  methodology,  data  requirements,  data  management  needs,  and 
analysis  requirements. 

4.5.4  Test  Plan 

The  testing  agent  that  will  be  executing  the  test  events  described  by  the  TEMP  will  develop  detailed  test 
plans  as  required  by  the  program  contract.  Service  policy,  or  the  Reference  (j) .  The  test  plan  is  the 
vehicle  that  translates  a  test  concept  and  statistical/analytical  test  design  into  concrete  resources, 
procedures,  and  responsibilities.  The  size  and  complexity  of  a  test  program  and  its  associated  test  plan 
are  determined  by  the  nature  of  the  system  being  tested  and  the  type  of  testing  that  is  to  be 
accomplished.  Some  major  weapons  systems  may  require  large  numbers  of  separate  tests  to  satisfy  test 
objectives  and,  thus,  require  a  multi-volume  test  plan;  other  testing  may  be  well-defined  by  a  relatively 
brief  test  plan.  The  test  plan  also  provides  a  description  of  the  equipment  configuration  and  known 
limitations  to  the  scope  of  testing.  An  example  of  typical  test  plan  format  is  shown  in  Figure  4-3. 
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Preliminary  Pages 

i  Title  Page 

ii.  Approval  Page 

iii-  Conditions  for  Release  of  Technical  Data 
iv.  Preface  (Abstract) 

V.  Executive  Summary 
vi-  Table  of  Contents* 

*The  actual  number  of  these  pages  wi  II  be  determined  by 
the  length  of  preliminary  elements  (e  g.,  Table  of  Contents, 
Terms  and  Abbreviations,  etc  ). 

Main  Body 

1.  Introduction 

2.  Background 

3.  Test  Item  Description 

4.  Overall  Test  Objectives 

5.  Constraints  and  Limitations 

6.  Test  Resources 

7.  Safety  Requirements 

8.  Security  Requirements 

9.  Test  Project  Management 

10.  Test  Procedures 

11.  Test  Reporting 

12.  Logistics 

13.  Environmental  Protection 
Annexes 

A.  Test  Condition  Matrix 

B.  Requirements  Traceability 

C.  Test  Information  Sheets 
D  Parameters  List 

E.  Data  Analysis  Plan 

F.  Instrumentation  Plan 

G.  Logistics  Support  Plan 
H  -Z  As  Required 

Li  St  of  Abbreviations 
Distribution 

Source:  AFFTC  Test  Plan  Preparation  Guide 


Figure  4-3  Example  of  Typical  Test  Plan  Format 


4.5.5  TRP 

The  Army  and  Air  Force  TRPs  are  examples  of  essential  test  planning  documents.  They  are  formal 
resource  documents  specifying  the  resources  required  to  support  the  test.  Because  the  TRPs  provide  the 
basis  for  fiscal  programming  and  coordinating  the  necessary  resources,  it  is  important  that  these 
documents  be  developed  in  advance  and  kept  current  to  reflect  maturing  resource  requirements  as  the 
test  program  develops.  The  Navy  makes  extensive  use  of  the  TEMP  to  document  T&E  resource 
requirements.  Each  Service  has  periodic  meetings  designed  to  review  resource  requirements  and 
resolve  problems  with  test  support. 
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4.5.6  Test  Reports 

4.5.6.1  Quick-Look  Reports 

Quick-look  reports  are  expeditious  analyses  performed  during  testing  using  limited  amounts  of  the 
database.  Such  analyses  often  are  used  to  assist  in  managing  test  operations.  Quick-look  reports  are 
used  occasionally  to  inform  higher  authorities  of  test  results.  Quick-look  reports  may  have  associated 
briefings  that  present  T&E  results  and  substantiate  conclusions  or  recommendations.  Quick-look  reports 
may  be  generated  by  the  contractor  or  Government  agency.  They  are  of  particularly  critical  interest  for 
high-visibility  systems  that  may  be  experiencing  some  development  difficulties.  Techniques  and  formats 
should  be  determined  before  the  start  of  testing  and  may  be  exercised  during  pretest  trials. 

4. 5. 6. 2  Final  Test  Report 

The  final  test  report  disseminates  the  test  information  to  decision  authorities,  program  office  staff,  and 
the  acquisition  community.  It  provides  a  permanent  record  of  the  execution  of  the  test  and  its  results. 
The  final  test  report  should  relate  the  test  results  to  the  critical  issues  and  address  the  objectives  stated 
in  the  test  design  and  test  plan.  A  final  test  report  may  be  separated  into  two  sections-a  main  section 
providing  the  essential  information  about  test  methods  and  results,  and  a  second  section  consisting  of 
supporting  appendixes  to  provide  details  and  supplemental  information.  The  type  of  information 
generally  included  in  the  final  test  report  is  shown  in  Figure  4-4. 
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1-  Test  purpose 
2  -  Issues  and  objectives 

3-  Method  of  accomplishment 

4-  Results  (keyed  to  the  objectives  and  issues) 

5  ~  Discussion,  conclusions,  and  recommendations 

Example  Appendixes: 

A  -  Detailed  test  description 

B  -  Test  environment 

C  -  Test  organization  and  operation 

D-  Instrumentation 

E  -  Data  collection  and  management 

F  -  Test  data 

G-  Data  analysis 

H-M&S 

I  -  RAM  information 
J-  Personnel 
K  -  Training 
L  -  Safety 
M  -  Security 
N  -  Funding 

O-  Post-test  Asset  disposition 


Figure  4-4  Types  of  Information  in  a  Final  Test  Report 
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The  final  test  report  may  contain  an  evaluation  and  analysis  of  the  results,  or  the  evaluation  may  be 
issued  separately.  The  analysis  tells  what  the  results  are,  whereas  an  evaluation  tells  what  the  results 
mean.  The  evaluation  builds  on  the  analysis  and  generalizes  from  it,  showing  how  the  results  apply 
outside  the  test  arena.  It  shows  what  the  implications  of  the  test  are  and  may  provide 
recommendations.  The  evaluation  may  make  use  of  independent  analyses  of  all  or  part  of  the  data;  it 
may  employ  data  from  other  sources  and  may  use  M&S  to  generalize  the  results  and  extrapolate  to 
other  conditions.  In  the  case  of  the  Army,  a  separate  system  evaluation  report  is  prepared  by 
independent  evaluators  within  the  AEC. 

4.6  Other  Test-Related  Status  Reports 

4.6.1  End-of-Test  Reports 

The  Services  are  required  by  Enclosure  6  of  Reference  (d)  to  submit  to  OSD  T&E  offices  copies  of  their 
formal  DT&E,  OT&E,  and  LFT&E  reports  that  are  prepared  at  the  end  of  each  period  of  testing  for  ACAT  I, 
lA,  and  oversight  programs.  These  reports  will  generally  be  submitted  in  a  timely  manner  to  permit  OSD 
review. 

4.6.2  BLRIP  Report 

Before  an  ACAT  I  or  DOT&E-designated  program  can  proceed  beyond  LRIP,  the  DOT&E  must  submit  a 
BLRIP  report  to  the  SecDef  and  the  Senate  and  House  of  Representatives  Committees  on  Armed 
Services,  National  Security,  and  Appropriations.  This  report  addresses  whether  the  OT&E  performed  was 
adequate  and  whether  the  lOT&E  results  confirm  that  the  items  or  components  tested  are  effective  and 
suitable  for  use  in  combat  by  typical  military  users.  The  report  may  include  information  on  the  results  of 
LFT&E  for  applicable  major  systems. 

4.6.3  LFT&E  Report 

Before  an  ACAT  I  or  DOT&E-designated  program  can  proceed  beyond  LRIP,  the  DOT&E  must  submit  an 
LFT&E  report  to  the  SecDef  and  the  Senate  and  House  of  Representatives  Committees  on  Armed 
Services,  National  Security,  and  Appropriations.  This  report  addresses  whether  the  LFT&E  performed 
was  adequate  and  whether  the  Service  LFT&E  results  confirm  that  the  items  or  components  tested  are 
lethal  or  vulnerable  considering  planned  use  in  combat.  The  report  may  be  included  with  the  DOT&E 
BLRIP  information  on  the  results  of  Service  lOT&E  for  applicable  MDAP  and  oversight  systems. 

4.6.4  AOTR 

See  page  50,  paragraph  3.9.9  of  this  guide. 

4.6.5  Annual  DT/SE  Report 

The  DASD(DT&E)  and  DASD(SE)  are  required  by  Reference  (w)  to  provide  the  SecDef,  the  USD(AT&L), 
and  Congress  with  an  annual  assessment  of  the  weapon  system  development  progress  for  programs  on 
the  OSD  T&E  oversight  list. 
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4.6.6  OA/EOA 

An  OA  is  an  evaluation  of  operational  effectiveness  and  suitability  made  by  an  independent  OTA,  with 
user  support  as  required,  on  other-than-production  systems.  An  OA  conducted  during  system 
integration  is  often  called  an  EOA. 

4.6.7  ILS 

ILS  is  a  unified  and  iterative  approach  to  the  management  and  technical  activities  needed  to  influence 
operational  and  materiel  requirements  and  design  specifications,  define  the  support  requirements  best 
related  to  system  design  and  to  each  other,  develop  and  acquire  the  required  support,  provide  required 
operational  support  at  lowest  cost,  seek  readiness  and  LCC  improvements  in  the  materiel  system  and 
support  systems  during  the  operational  life  cycle,  and  repeatedly  examine  support  requirements 
throughout  the  service  life  of  the  system. 

4.6.8  Exit  Criteria 

Exit  criteria  are  program-specific  accomplishments  that  must  be  satisfactorily  demonstrated  before  a 
program  can  progress  further  in  the  current  acquisition  phase  or  transition  to  the  next  acquisition 
phase.  Exit  criteria  are  normally  selected  to  track  progress  in  important  technical,  schedule,  or 
management  risk  areas.  They  serve  as  gates  that,  when  successfully  passed  or  exited,  demonstrate  that 
the  program  is  on  track  to  achieve  its  final  program  goals  and  should  be  allowed  to  continue  with 
additional  activities  within  an  acquisition  phase  or  be  considered  for  continuation  into  the  next 
acquisition  phase.  Exit  criteria  are  some  level  of  demonstrated  performance  outcome  (e.g.,  level  of 
engine  thrust),  the  accomplishment  of  some  process  at  some  level  of  efficiency  (e.g.,  manufacturing 
yield),  or  successful  accomplishment  of  some  event  (e.g.,  first  flight),  or  some  other  criterion  (e.g., 
establishment  of  a  training  program  or  inclusion  of  a  particular  clause  in  the  follow-on  contract)  that 
indicates  that  aspect  of  the  program  is  progressing  satisfactorily.  Exit  criteria  are  documented  in  the 
ADM. 

4.7  Summary 

A  wide  range  of  documentation  is  available  to  the  test  manager  and  should  be  used  to  develop  T&E 
programs  that  address  all  relevant  issues.  The  PM  must  work  to  ensure  that  T&E  requirements  are 
considered  at  the  outset  when  the  acquisition  strategy  and  funding  requirements  are  formulated.  The 
PM  must  also  require  early,  close  coordination  and  a  continuing  dialogue  among  those  responsible  for 
integration  of  functional  area  planning  and  the  TEMP.  Service-level  programs  have  their  own  unique 
documentation  requirements. 
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Test  &  Evaluation  Management  Guide 
Chapter  5  -  Test  &  Evaluation 

5.1  Introduction 

This  chapter  describes  the  evaiuation  portion  of  the  T&E  process,  it  stresses  the  importance  of 
estabiishing  and  maintaining  a  ciear  audit  traii  from  system  requirements  through  COis,  evaiuation 
criteria,  test  objectives,  and  MOEs  to  the  actuai  evaiuation  process  itseif.  The  importance  of  the  use  of 
data  from  aii  sources  is  discussed  as  are  the  differences  in  approaches  to  evaiuating  technicai  and 
operationai  data. 

5.2  "Test"  and  "Evaluation" 

Test :  Test  denotes  any  program  or  procedure  that  is  designed  to  obtain,  verify,  or  provide  data  for  the 
evaluation  of  any  of  the  following:  (1)  progress  in  accomplishing  developmental  objectives;  (2)  the 
performance,  operational  capability,  and  suitability  of  systems,  subsystems,  components,  and 
equipment  items;  and  (3)  the  vulnerability  and  lethality  of  systems,  subsystems,  components,  and 
equipment  items. 

Evaluation  :  Evaluation  denotes  the  process  whereby  data  are  logically  assembled,  analyzed,  and 
compared  to  expected  performance  to  aid  in  systematic  decision  making.  It  may  involve  review  and 
analysis  of  qualitative  or  quantitative  data  obtained  from  design  reviews,  hardware  inspections,  M&S, 
hardware  and  software  testing,  metrics  review,  and  operational  usage  of  equipment. 

Test  and  Evaluation  :  T&E  is  a  process  by  which  a  system  or  components  are  tested  and  results  analyzed 
to  provide  performance  related  information.  This  information  has  many  uses,  including  risk 
identification  and  mitigation  as  well  as  providing  empirical  data  to  validate  models  and  simulations.  T&E 
enables  an  assessment  of  the  attainment  of  technical  performance,  specifications,  and  system  maturity 
to  determine  whether  systems  are  operationally  effective,  suitable,  and  survivable  for  their  intended 
use.  There  are  three  distinct  types  of  T&E  defined  in  statute  or  regulation:  Developmental  Test  and 
Evaluation  (DT&E),  Operational  Test  and  Evaluation  (OT&E),  and  Live  Fire  Test  and  Evaluation  (LFT&E). 
These  are  all  covered  in  subsequent  chapters  of  this  guide. 

5.3  The  Evaluation  Process 

As  illustrated  in  Figure  5-1,  the  evaluation  process  requires  a  broad  analytical  approach  with  careful 
focus  on  the  development  of  an  overall  test  plan  that  will  provide  timely  answers  to  COIs  and  questions 
required  by  decision  authorities  throughout  all  the  acquisition  phases.  Evaluations  should  focus  on  KPPs; 
i.e.,  those  minimum  attributes  or  characteristics  considered  most  essential  for  an  effective  military 
capability.  The  DAU  Glossary  of  Defense  Acquisition  Acronyms  &  Terms  (Reference  (ah))  defines  an 
attribute  as  a  quantitative  or  qualitative  characteristic  of  an  element  or  its  actions 

A  functional  block  diagram  of  a  generic  evaluation  process  is  also  shown  in  Figure  5-1.  The  process 
begins  with  the  identification  of  a  deficiency  or  need  and  the  documentation  of  an  operational 
requirement.  It  continues  with  the  identification  of  COIs  that  must  be  addressed  to  determine  the 
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degree  to  which  the  system  meets  user  requirements.  Objectives  and  thresholds  must  then  be 
established  to  define  required  performance  or  supportability  parameters  and  to  evaluate  progress  in 
reaching  them.  Data  requirements  are  then  consolidated/integrated  in  the  TEMP  evaluation  framework 
to  identify  distribution  of  testing  activities  necessary  to  support  evaluations.  T&E  analysts  then 
decompose  the  issues  into  measurable  test  elements,  conduct  the  necessary  testing,  review  and  analyze 
the  test  data,  weigh  the  test  results  against  the  evaluation  criteria,  and  prepare  an  evaluation  report  for 
the  decision  authorities. 


Figure  5-1  Illustrative  Generic  Evaluation  Process 
5.4  Distinction  Between  Issues  and  Criteria 

Issues,  as  defined  in  Reference  (ah) ,  are  questions  regarding  a  system  that  require  answers  during  the 
acquisition  process.  Those  answers  may  be  needed  to  aid  in  the  development  of  an  acquisition  strategy, 
to  refine  performance  requirements  and  designs,  or  to  support  milestone  decision  reviews.  As  defined 
in  Reference  (ah) ,  Evaluation  criteria  are  the  standards  by  which  accomplishments  of  required  technical 
and  operational  effectiveness  and/or  suitability  characteristics  or  resolution  of  operational  issues  may 
be  assessed. 


77 


525  &-yR9l3&lzuSy'a  l-yt-ESrSyuDcms 


/KI-L0Jji 


The  evaluation  program  may  be  constructed  using  a  structured  approach  identifying  each  issue. 

For  example: 

•  Issue:  a  statement  of  the  question  to  be  answered. 

•  Scope:  detailed  conditions  and  range  of  conditions  that  will  guide  the  T&E  process  for  this  issue. 

•  Criteria:  quantitative  or  qualitative  standards  that  will  answer  the  issue. 

•  Rationale:  full  justification  to  support  the  selected  criteria. 

A  formal  evaluation  plan  is  typically  created  to  support  the  overall  evaluation  effort.  These  plans  vary  by 
Service  and  Defense  Agency,  see  Figure  5-2  for  an  example  of  key  topics  in  an  evaluation  plan. 


•  Introduction 

•  Purpose 

•  Scope 

•  Background 

•  System  Description 

•  GDIs  and  Criteria 

•  Projected  Threat 

•  Test  and  Evaluation  Milestones 

•  Evaluation  Strategy 

•  Evaluation  Concept 

•  Operational  Effectiveness 

Issue  1 

Scope 

Criteria 

Rationale 

Evaluation  Approach 

Analysis  of  MOPs  and  Data  Presentations 
MOP  1  through  MOP  "x" 

Issue  2  through  Issue  "n" 

•  Operational  Suitability 

Issue  "n+l"  through  Issue  "n+x" 

•  Data  Source  Matrix 

•  Description  of  other  Primary  Data  Sources 

•  Test  Approach 

Test  Scope 

Factors  and  Conditions 

Sample  Size  and  Other  Test  Design  Considerations 
Data  Authentication 

•  Evaluation  Database  Structure 

Identification  of  Required  Files 
Description  of  File  Relationships 
Data  Element  Definitions 


Figure  5-2  Generic  Example  of  Key  Topics  in  an  Evaluation  Plan 


5.4.1  KPPs/COIs 
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The  KPPs  often  can  support  the  development  of  a  hierarchy  of  COIs  and  less  significant  issues.  COIs  are 
those  questions  relating  to  a  systems  operational,  technical,  support,  or  other  capability.  These  issues 
must  be  answered  before  the  systems  overall  worth  can  be  estimated/evaluated,  and  they  are  of 
primary  importance  to  the  decision  authority  in  allowing  the  system  to  advance  to  the  next  acquisition 
phase. 

COIs  in  the  TEMP  may  be  derived  from  the  KPPs  found  in  the  capability  requirements  documents-CDD 
and  CPD.  The  system  requirements  and  baseline  documentation  will  provide  many  of  the  performance 
parameters  required  to  develop  the  hierarchy  of  issues. 

5.4.2  Evaluation  Issues 

Evaluation  issues  are  those  issues  addressed  in  technical  or  operational  evaluations  during  the 
acquisition  process.  Evaluation  issues  can  be  separated  into  technical  or  operational  issues  and  are 
addressed  in  the  TEMP. 

Technical  issues  primarily  concern  technical  parameters  or  characteristics  and  engineering  specifications 
normally  assessed  and  verified  as  part  of  DT.  Operational  issues  concern  effectiveness  and  suitability 
characteristics  for  functions  to  be  performed  by  equipment/personnel.  Operational  issues  pertain  to 
validating  the  systems  operational  performance  when  examined  in  a  realistic  operational  mission 
environment.  Both  technical  and  operational  issues  should  take  joint  operating  environments  into 
consideration. 

5.4.3  Test  Issues 

Test  issues  are  a  subset  of  evaluation  issues  and  address  areas  of  uncertainty  that  require  test  data  to 
resolve  the  issue  adequately.  Test  issues  may  be  partitioned  into  technical  issues  that  are  addressed  by 
the  DT&E  community  and  operational  issues  that  are  addressed  by  the  OT&E  community.  Test  issues 
may  be  further  divided  into  critical  and  noncritical  categories.  All  critical  T&E  issues,  objectives, 
methodologies,  and  evaluation  criteria  should  be  defined  during  the  initial  establishment  of  an 
acquisition  program.  COIs  are  documented  in  the  TEMP.  These  evaluation  issues  serve  to  define  the 
testing  required  for  each  phase  of  the  acquisition  process  and  serve  as  the  structure  to  guide  the  testing 
program  so  these  data  may  be  compared  against  performance  criteria. 

5.4.4  Criteria 

Criteria  are  statements  of  a  systems  required  technical  performance  and  operational  effectiveness  and 
suitability.  Criteria  are  often  expressed  as  objectives  and  thresholds.  These  performance  measurements 
provide  the  basis  for  collecting  data  used  to  evaluate/answer  test  issues.  Criteria  must  be  unambiguous 
and  assessable  whether  stated  qualitatively  or  quantitatively.  For  example,  they  may  compare  the 
mission  performance  of  the  new  system  to  the  one  being  replaced,  compare  the  new  system  to  a 
predetermined  standard,  or  compare  a  system  to  a  similar  system.  As  such,  they  should  be  developed  in 
close  coordination  with  the  system  user,  other  testers,  and  specialists  in  all  other  areas  of  operational 
effectiveness  and  suitability.  These  values  may  be  changed  as  systems  develop  and  associated  testing 
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and  evaluation  proceed.  Every  issue  should  have  at  least  one  criterion  that  is  a  concise  measure  of  the 
function.  Values  must  be  realistic  and  achievable  within  the  state  of  the  art  of  engineering  technology.  A 
quantitative  or  qualitative  criterion  should  have  a  clear  definition,  free  of  ambiguous  or  imprecise 
terminology,  such  as  adequate,  sufficient,  or  acceptable. 

A  threshold  for  a  performance  parameter  lists  a  minimum  acceptable  operational  value  below  which  the 
utility  of  the  system  becomes  questionable  (See  Reference  (ah) ).  Thresholds  are  stated  quantitatively 
whenever  possible.  Specification  of  minimally  acceptable  performance  in  measurable  parameters  is 
essential  to  selecting  appropriate  MOEs,  which  in  turn  heavily  influence  test  design.  Thresholds  are  of 
value  only  when  they  are  testable;  i.e.,  actual  performance  can  be  measured  against  them.  The  function 
of  T&E  is  to  verify  the  attainment  of  required  thresholds.  The  PM  must  budget  for  the  attainment  of 
threshold  values. 

Objectives  are  the  desired  operational  goal  associated  with  a  performance  attribute,  beyond  which  any 
gain  in  utility  does  not  warrant  additional  expenditure.  The  objective  value  may  be  an  operationally 
significant  increment  above  the  threshold.  An  objective  value  as  defined  by  Reference  (ah)  may  be  the 
same  as  the  threshold  when  an  operationally  significant  increment  above  the  threshold  is  not  significant 
or  useful.  Objectives  are  not  normally  addressed  by  the  operational  tester,  whose  primary  concern  is  to 
determine  if  thresholds  in  the  approved  CPD  and  COIs  have  been  satisfied  (See  Reference  (d) ). 

Going  into  system  demonstration,  thresholds  and  objectives  are  expanded  along  with  the  identification 
of  more  detailed  and  refined  performance  capabilities  and  characteristics  resulting  from  trade-off 
studies  and  testing  conducted  during  the  evaluation  of  EDMs.  Along  with  the  CPD,  thresholds  and 
objectives  should  remain  relatively  stable  through  production.  Testers  should: 

Review  requirements  as  they  are  developed  to  assess  whether  they  are  unambiguous,  testable,  relevant 
to  accomplishing  missions  in  combat,  and  operationally  and  technically  realistic. 

Seek  opportunities  to  be  involved  in  reviews  of  requirements  conducted  before  those  requirements  are 
submitted  for  consideration  within  OSD. 

For  each  project  under  oversight,  review  the  TES  and  TEMP  to  ensure  that  they  include  testing  in 
realistic  operational  environments  initiated  during  development  and  continuing  through  OT.  This 
continuum  of  realistic  testing  will  place  increasing  stress  on  subsystem  components  before  final 
integration  into  a  full-up  system,  thereby  identifying  problems  when  they  can  be  fixed  cost-effectively. 

Identify  operational  concerns  to  program  offices  and  DOT&E  leadership  at  the  earliest  possible  time  so 
that  they  can  be  resolved  in  a  timely  manner. 

Identify  for  action  by  DOT&E  leadership  test-critical  resource  shortfalls. 

Ensure  that  testing  in  a  joint  environment  is  included  in  TESs  and  TEMPs  as  appropriate  and  feasible. 

Ensure  that  developers  and  the  operational  community  share  a  clear,  common  understanding  of  the 
planned  concept  of  operations  (CONOPS)  or  identify  for  action  by  DOT&E  leadership  inconsistencies  in 
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those  views.  If  the  CONORS  is  not  available,  work  to  ensure  that  a  representative  set  of  CONORS  is 
included  in  TESs  and  TEMRs. 

5.5  MOEs 

Requirements,  thresholds,  and  objectives  established  in  early  program  documentation  form  the  basis 
for  evaluation  criteria.  If  program  documentation  is  incomplete,  the  tester  may  have  to  develop 
evaluation  criteria  in  the  absence  of  specific  requirements.  Evaluation  criteria  are  associated  with 
objectives,  sub-objectives,  KRRs,  and  MOEs/MOSs  and  ultimately  devolved  into  MORs.  MOE  refers  to 
the  effectiveness  of  a  solution  but  is  independent  of  any  particular  solution,  while  an  MOR  refers  to  the 
actual  performance  of  a  specific  entity  (selected  solution).  Thus,  an  MOE  can  be  used  to  validate  that 
the  system  meets  the  user's  intended  needs/mission  objectives,  and  an  MOR  can  be  used  to  verify  that 
the  system  meets  the  user's  stated  requirements.  Reference  (ah)  further  defines  the  key  terms  below: 

MOE  :  The  data  used  to  measure  the  military  effect  (mission  accomplishment)  that  comes  from  the  use 
of  the  system  in  its  expected  environment.  That  environment  includes  the  system  under  test  and  all 
interrelated  systems,  that  is,  the  planned  or  expected  environment  in  terms  of  weapons,  sensors, 
command  and  control  (C2),  and  platforms,  as  appropriate,  needed  to  accomplish  an  end-to-end  mission 
in  combat).  See  Operational  Effectiveness  (OE),  Measure  of  Rerformance  (MOR),  Operational  Suitability 
(OS),  and  Measure  of  Suitability  (MOS). 

MOS  :  Measure  of  an  items  ability  to  be  supported  in  its  intended  operational  environment.  MOSs 
typically  relate  to  readiness  or  operational  availability  and,  hence  reliability,  maintainability,  and  the 
items  support  structure. 

MOP  :  System-particular  performance  parameters  such  as  speed,  payload,  range,  time-on-station, 
frequency,  or  other  distinctly  quantifiable  performance  features.  Several  MORs  may  be  related  to 
achieving  a  particular  Measure  of  Effectiveness  (MOE). 

MOEs  (and  MOSs)  are  important  because  they  determine  how  test  results  will  be  judged,  and  because 
test  planning  is  directed  toward  obtaining  these  measures,  it  is  important  that  they  be  defined  early.  For 
example,  an  MOE  (airspeed)  may  have  an  associated  evaluation  criterion  (450  knots)  against  which  the 
actual  performance  (425  knots)  is  compared  to  arrive  at  a  rating.  Generally,  the  resolution  of  each 
critical  issue  is  in  terms  of  the  evaluation  of  some  MOE.  In  this  case,  the  operating,  implementing,  and 
supporting  commands  must  agree  with  the  criteria  before  the  test  organization  makes  use  of  them  in 
assessing  test  results.  Ensuring  that  MOEs  can  be  related  to  the  user's  operational  requirements  is  an 
important  consideration  when  identifying  and  establishing  evaluation  criteria. 

Testers  must  ensure  that  evaluation  criteria  and  MOEs  are  updated  if  requirements  change.  Testers 
must  determine  mission  tasks  and  system-of-systems  performance  in  order  to  establish  links  to  mission 
effectiveness  measures.  Mission  end  states  and  desired  effects  must  be  predetermined  in  order  to 
derive  MOEs  used  in  system-of-systems  testing.  The  MOEs  should  be  so  specific  that  the  systems 
effectiveness  during  developmental  and  operational  testing  can  be  assessed  using  some  of  the  same 
effectiveness  criteria  as  the  AoAs. 
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5.6  Evaluation  Planning 

5.6.1  Evaluation  Planning  Techniques 

Techniques  that  have  been  proven  effective  in  evaluation  planning  include  process  analysis,  design  or 
engineering  analysis,  matrix  analysis,  and  dendritic  analysis. 

Evaluation  planning  is  an  iterative  process  that  requires  formal  and  informal  analyses  of  system 
operation  (e.g.,  threat  environment,  system  design,  tactics,  and  interoperability).  Recognized  analytical 
techniques  (including  computer  models)  are  used  to  interpret  or  explain  the  behavior/performance  of 
the  system  element  during  T&E.  Analysis  of  test  data  or  review  and  analysis  of  design  data  will  be  used 
as  appropriate  to  verify  the  systems  requirements.  Evaluation  planning  attempts  to  identify  in  advance 
exactly  what  is  to  be  evaluated,  how  the  data  are  to  be  collected,  and  at  least  a  qualitative 
understanding  of  how  the  data  will  be  analyzed. 

5. 6. 1.1  Process  Analysis  Techniques 

Process  analysis  techniques  consist  of  thinking  through  how  the  system  will  be  used  in  a  variety  of 
environments,  threats,  missions,  and  scenarios  in  order  to  understand  the  events,  actions,  situations, 
and  results  that  are  expected  to  occur.  This  technique  aids  in  the  identification  and  clarification  of 
appropriate  MOEs,  test  conditions,  and  data  requirements. 

5. 6. 1.2  Design/Engineering  Analysis  Techniques 

Design  or  engineering  analysis  techniques  are  used  to  examine  all  mechanical  or  functional  operations 
that  the  system  has  been  designed  to  perform.  These  techniques  involve  a  systematic  exploration  of  the 
systems  hardware  and  software  components,  purpose,  performance  bounds,  manpower  and  personnel 
considerations,  known  problem  areas,  and  impact  on  other  components.  Exploration  of  the  way  a 
system  operates  compared  to  intended  performance  functions  often  identifies  issues,  MOEs,  specific 
data,  test  events,  and  required  instrumentation. 

5. 6. 1.3  Matrix  Analysis  Techniques 

Matrix  analysis  techniques  are  useful  for  analyzing  any  situation  in  which  two  classifications  must  be 
cross-referenced.  For  example,  a  matrix  of  types  of  data  versus  means  of  data  collection  can  reveal  not 
only  types  of  data  having  no  planned  means  of  collection,  but  also  redundant  or  backup  collection 
systems.  Matrix  techniques  are  useful  as  checklists,  as  organizational  tools,  or  as  a  way  of  identifying 
and  characterizing  problem  areas.  Matrix  techniques  are  effective  for  tracing  a  system's  operational 
requirements  through  contractual  specification  documents,  issues,  and  criteria  to  sources  of  individual 
data  or  specific  test  events. 

5. 6. 1.4  Dendritic  Analysis  Techniques 

Dendritic  analysis  is  an  effective  way  of  decomposing  COIs  to  the  point  where  actual  data  requirements 
and  test  measurements  can  be  identified.  In  these  techniques,  issues  are  successively  broken  down  into 
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objectives,  MOEs,  MOPs,  and  data  requirements  in  a  rootlike  structure  as  depicted  in  Figure  5-3.  In  this 
approach,  objectives  are  used  to  clearly  express  the  broad  aspects  of  T&E  related  to  the  COIs  and  the 
overall  purpose  of  the  test.  The  MOEs  are  developed  as  subsets  of  the  objectives  and  are  designed  to 
treat  specific  and  addressable  parts  of  the  objectives.  Each  MOE  is  traceable  as  a  direct  contributor  to 
one  objective  and,  through  it,  is  identifiable  as  a  direct  contributor  to  addressing  one  or  more  COIs.  Each 
test  objective  and  its  related  MOE(s)  is  also  linked  to  one  or  more  MOPs  (quantitative  or  qualitative 
measures  of  system  performance  under  specified  conditions)  that  in  turn  are  tied  to  specific  data 
elements.  Data  elements  are  observations  and/or  measurements  under  specified  conditions. 
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Figure  5-3  Example  of  Dendritic  Analysis  Showing  Critical  Issue  Decomposition 
Sources  of  Data 

Data  collection  requirements  should  be  considered  throughout  the  life  cycle  of  the  system.  As  systems 
move  beyond  DT,  collecting  required  data  becomes  increasingly  difficult  because  the  data  collection 
hooks  in  the  system  software  are  often  removed.  Additional  cost  and  schedule  delays  can  occur  if 
software  builds  need  to  be  revised  to  collect  the  data  required  for  subsequent  T&E.  In  addition,  the 
evaluators  need  to  ensure  that  the  appropriate  hardware  and  software  are  available  to  perform  data 
reduction  and  analysis.  These  tools  need  to  keep  pace  with  system  maturity  in  parallel  rather  than 
serially  to  maintain  schedule;  however,  these  support  systems  are  usually  afterthoughts. 
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As  evaluation  and  analysis  planning  matures,  focus  turns  toward  identifying  data  sources  as  a  means  for 
obtaining  each  data  element.  Initial  identification  tends  to  be  generic  such  as  engineering  study, 
simulation,  modeling,  or  contractor  test.  Later  identification  reflects  specific  studies,  models,  and/or 
tests.  A  data  source  matrix  is  a  useful  planning  tool  to  show  where  data  are  expected  to  be  obtained 
during  the  T&E  of  the  system.  Many  sources  of  data  can  contribute  to  the  evaluation.  Principal  sources 
include  studies  and  analyses,  models,  simulations,  war  games,  contractor  testing,  DT,  OT,  and 
comparable  systems. 

5.7  Evaluating  Developmental  And  Operational  Tests 

Technical  and  operational  evaluations  employ  different  techniques  and  have  different  evaluation 
criteria.  DT&E  is  primarily  a  technical  verification  evaluation  while  OT&E  addresses  and  validates  the 
operational  aspects  of  a  system.  Technical  evaluation  deals  primarily  with  instrumented  tests  and 
statistically  valid  data.  An  operational  evaluation  deals  with  operational  realism  and  combat 
uncertainties.  DT&E  uses  technical  criteria  for  evaluating  system  performance.  These  criteria  are  usually 
parameters  that  can  be  measured  during  controlled  DT&E.  They  are  particularly  important  to  the 
developing  organization  and  the  contractor  but  are  of  less  interest  to  the  independent  operational 
tester.  The  operational  tester  focuses  on  issues  such  as  demonstrating  target  acquisition  at  useful 
ranges,  air  superiority  in  combat,  or  the  probability  of  accomplishing  a  given  mission.  An  integrated  test 
strategy  strives  to  ensure  collaborative  test  planning  and  execution  of  developmental  (both  contractor 
and  Government)  and  operational  test  events  to  provide  shared  data  in  support  of  independent 
analysis,  evaluation,  and  reporting  by  all  stakeholders. 

For  example,  during  DT&E,  firing  may  be  conducted  on  a  round-by-round  basis  with  each  shot  designed 
to  test  an  individual  specification  or  parameter  with  other  parameters  held  constant.  Such  testing  is 
designed  to  measure  the  technical  performance  of  the  system.  In  contrast,  in  OT&E,  technical 
performance  regarding  individual  specifications/parameters  is  de-emphasized  and  the  environment  is 
less  controlled.  The  OT&E  authority  must  assess  whether,  given  this  technical  performance,  the  weapon 
system  is  operationally  effective  and  operationally  suitable  when  employed  under  realistic  combat  (with 
opposing  force)  and  environmental  conditions  by  typical  unit  personnel.  The  emphasis  in  DT  is  strictly  on 
the  use  of  quantitative  data  to  verify  attainment  of  technical  specifications. 

Quantitative  data  are  usually  analyzed  using  some  form  of  statistics.  Qualitative  data  take  on  increasing 
importance  in  QT&E  when  effectiveness  and  suitability  issues  are  being  explored.  Many  techniques  are 
used  to  analyze  qualitative  data.  They  range  from  converting  expressions  of  preference  or  opinion  into 
numerical  values  to  establishing  a  consensus  by  committee.  For  example,  a  committee  may  assign 
values  to  parameters  such  as  feel,  ease  of  use,  friendliness  to  the  user,  and  will  the  user  want  to  use  it, 
on  a  scale  of  1  to  10.  Flowever,  care  should  be  exercised  in  the  interpretation  of  the  results  of  qualitative 
evaluations.  When  numbers  are  assigned  to  average  qualitative  evaluations  and  their  related  statistical 
measures,  meanings  and  interpretation  will  differ  from  statistical  measures  used  to  characterize 
quantitative  data. 
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5.7.1  Technical  Evaluation 

The  Services  materiel  development  organizations  are  usually  responsible  for  oversight  of  all  aspects  of 
DT&E  including  the  technical  evaluation.  The  objectives  of  a  technical  evaluation  include  the  following: 

•  To  assist  the  developers  by  providing  information  relative  to  technical  performance, 
qualification  of  components,  compatibility,  interoperability,  vulnerability,  lethality, 
transportability,  RAM,  manpower  and  personnel,  system  safety,  ILS,  correction  of  deficiencies, 
accuracy  of  environmental  documentation,  and  refinement  of  requirements. 

•  To  ensure  the  effectiveness  of  the  manufacturing  process  of  equipment  and  procedures  through 
production  qualification  testing. 

•  To  confirm  readiness  for  lOT&E  by  ensuring  that  the  system  is  stressed  beyond  the  levels 
expected  in  the  operational  environment. 

•  To  provide  information  to  the  decision  authority  at  each  decision  point  regarding  a  systems 
technical  performance  and  readiness  to  proceed  to  the  next  phase  of  acquisition. 

5.7.2  Operational  Evaluation 

The  independent  OT&E  authority  is  responsible  for  the  operational  evaluation.  The  objectives  of  an 
operational  evaluation  include  the  following: 

•  To  assist  the  developers  by  providing  information  relative  to  operational  performance;  doctrine, 
training,  logistics,  and  tactics,  techniques  and  procedures  (TTP);  safety;  survivability;  manpower; 
technical  publications;  RAM;  correction  of  deficiencies;  accuracy  of  environmental 
documentation;  and  refinement  of  requirements. 

•  To  assist  decision  makers  in  ensuring  that  only  systems  that  are  operationally  effective  and 
suitable  are  delivered  to  the  operating  forces. 

•  To  provide  information  to  the  decision  authority  at  each  decision  point  as  to  a  systems 
operational  effectiveness,  suitability,  and  readiness  to  proceed  to  the  next  phase  of  acquisition. 

•  To  assess,  from  the  user's  viewpoint,  a  system's  desirability,  considering  capabilities  of  systems 
already  fielded,  and  the  benefits  or  burdens  associated  with  system  support  in  an  operational 
environment. 

•  To  determine  the  systems  operability  in  the  required  climatic  and  realistic  battlefield 
environments  to  include  natural,  induced,  and  countermeasure  environments. 

5.8  Summary 

A  primary  consideration  in  identifying  information  to  be  generated  by  an  evaluation  program  is  having  a 
clear  understanding  of  the  decisions  the  information  will  support.  The  importance  of  structuring  the  T&E 
program  to  support  the  resolution  of  COIs  cannot  be  overemphasized.  It  is  the  responsibility  of  those 
involved  in  the  evaluation  process  to  ensure  that  the  proper  focus  is  maintained  on  key  issues,  the  T&E 
program  yields  information  on  critical  technical  and  operational  issues,  all  data  sources  necessary  for  a 
thorough  evaluation  are  tapped,  and  evaluation  results  are  communicated  in  an  effective  and  timely 
manner.  The  evaluation  process  should  be  evolutionary  throughout  the  acquisition  phases. 
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Test  &  Evaluation  Management  Guide 
Chapter  6  -  Introduction  to  DT&E 

6.1  Introduction 

Materiel  acquisition  includes  the  iterative  process  of  designing,  building,  testing,  identifying  deficiencies, 
fixing,  retesting,  and  repeating  as  needed  to  completion.  DT&E  is  an  important  aspect  of  this  process. 
DT&E  is  performed  in  the  factory,  in  the  laboratory,  and  on  the  proving  ground.  It  is  conducted  by 
subcontractors  as  they  are  developing  the  components  and  subassembly;  by  the  prime  contractor  as  the 
components  are  assembled  and  integration  of  the  system  is  ensured;  and  by  the  Government  to  verify 
and  demonstrate  how  well  the  weapon  system  meets  its  technical  requirements. 

As  defined  in  Reference  (ah) ,  DT&E  is  (1)  Any  testing  used  to  assist  in  the  development  and  maturation 
of  products,  product  elements,  or  manufacturing  or  support  processes.  (2)  Any  engineering-type  test 
used  to  verify  status  of  technical  progress,  verify  that  design  risks  are  minimized,  substantiate 
achievement  of  contract  technical  performance,  and  certify  readiness  for  initial  OT.  Development  tests 
generally  require  instrumentation  and  measurements  and  are  accomplished  by  engineers,  technicians, 
or  soldier  operator-maintainer  test  personnel  in  a  controlled  environment  to  facilitate  failure  analysis. 

DT&E  is  conducted  to  demonstrate  that  the  engineering  design  and  development  process  is  complete.  It 
is  used  by  the  contractor  to  reduce  risk,  validate  and  qualify  the  design,  and  ensure  that  the  product  is 
ready  for  Government  acceptance.  DT&E  results  are  evaluated  to  ensure  that  design  risks  have  been 
minimized  and  that  the  system  will  meet  specifications.  The  results  are  also  used  to  estimate  the 
systems  military  utility  when  it  is  introduced  into  service.  DT&E  serves  a  critical  purpose  in  reducing  the 
risks  of  development  by  testing  all  components  and  subsystems,  with  emphasis  on  those  with  high  risks. 
Finally,  DT&E  is  the  Government  developing  agency's  tool  used  to  verify  that  the  system  performs  as 
technically  specified  and  that  the  system  is  ready  for  OT.  Using  various  illustrative  systems,  this  chapter 
provides  a  general  discussion  of  contractor  and  Government  DT&E  activities,  stresses  the  need  for  an 
integrated  test  program,  describes  some  special-purpose  DTs,  and  discusses  several  factors  that  may 
influence  the  extent  and  scope  of  the  DT&E  program. 

DT&E  is  conducted  throughout  the  system  life  cycle  from  program  initiation  through  system 
sustainment  in  order  to  reduce  design  and  programmatic  risks  and  to  provide  assessments.  DT&E  may 
be  a  mix  of  contractor  testing  and  Government  testing.  A  DT&E  program: 

•  Verifies  achievement  of  CTPs  and  KSAs,  and  assesses  progress  toward  achievement  of  COIs. 

•  Verifies  that  the  system  satisfies  the  thresholds  prescribed  in  the  capabilities  documents. 

•  Provides  data  to  the  PM  to  enable  root  cause  determination  and  to  identify  corrective  actions. 

•  Assesses  system  readiness  for  lOT&E. 

•  Characterizes  system  functionality. 

•  Provides  information  for  cost,  performance,  and  schedule  trade-offs. 

•  Assesses  system  specification  compliance. 

•  Reports  on  program  progress  to  plan  for  reliability  growth  and  characterizes  reliability  and 
maintainability  for  use  during  key  reviews. 
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•  Identifies  system  capabilities,  limitations,  and  deficiencies. 

•  Assesses  system  safety. 

•  Assesses  compatibility  with  legacy  systems. 

•  Stresses  the  system  within  an  intended  mission  environment. 

•  Supports  information  assurance  certification  and  accreditation. 

•  Supports  the  joint  interoperability  certification  processes. 

•  Documents  achievement  of  contractual  technical  performance,  and  verify  incremental 
improvements  and  system  corrective  actions. 


The  PM  should  develop  a  robust,  integrated  T&E  strategy  for  DT&E,  OT&E,  and  LFT&E  to  validate  system 
performance  and  ensure  that  the  product  provides  measurable  improvement  to  operational  capabilities. 
The  integrated  approach,  however,  should  not  compromise  DT&E,  OT&E,  or  LFT&E  objectives. 

6.2  DT&E  and  The  System  Acquisition  Cycle 

As  illustrated  in  Figure  6-1,  DT&E  focuses  on  the  technological  and  engineering  aspects  of  a  system  or 
piece  of  hardware  or  software  and  is  revisited  as  systems  evolve  or  are  upgraded.  DT&E  may  begin 
before  program  initiation  with  the  evaluation  of  evolving  technology,  and  it  continues  after  the  system  is 
in  the  field.. 
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Figure  6-1  Example  of  Testing  Activities  throughout  the  Acquisition  Lifecycle 
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6.2.1  DT&E  Prior  to  MSA 

Prior  to  program  initiation,  M&S,  analytical  studies,  and  technology  feasibility  testing  may  be  conducted 
to  confirm  that  the  technologies  being  considered  for  the  proposed  system  are  appropriate,  are 
technically  feasible,  and  have  potential  for  needed  maturation. 

6.2.2  DT&E  During  MSA  and  TD  Phases 

During  these  phases,  the  testing  conducted  depends  on  the  state  of  technical  maturity  of  the  test 
articles  design.  DT&E  that  takes  place  is  conducted  by  a  contractor  or  the  Government  to: 

•  Assist  in  selecting  the  preferred  alternative  of  system  concepts,  technologies,  and  designs. 

•  Influence  subsystem  and  system  design  updates. 

•  Support  updated  AoA. 

•  Contribute  to  requirements  updates. 

Government  testers  participate  in  this  testing  because  information  obtained  can  be  used  to  support 
technical  reviews,  early  test  planning  in  the  TD  phase,  and  development  of  the  CDD  and  the  RFP  for  MS 
B.  The  information  obtained  from  these  tests  may  also  be  used  to  support  a  program  start  decision  by 
the  Services  or  OSD.  DT&E  during  the  TD  phase  often  supports  a  down-select  decision  when  considering 
multiple  competitors  for  the  subsequent  EMD  phase.  Multiple  decision  points  in  the  TD  phase  are 
supported  by  DT&E  such  as  Preliminary  Design  Review  (PDR),  AoA  completion,  down-select,  and  CDD 
completion. 

6.2.3  DT&E  During  EMD  Phase 

DT  is  used  to  demonstrate  that  technical  risk  areas  have  been  identified  and  can  be  reduced  to 
acceptable  levels;  the  best  technical  approach  can  be  identified;  and,  from  this  point  on,  engineering 
efforts  will  be  required  rather  than  experimental  efforts.  It  supports  the  decision  review  that  considers 
transition  from  prototype  design  into  advanced  engineering  and  construction  of  the  EDM.  This  DT&E 
includes  contractor/Government  integrated  testing,  engineering  design  testing,  and  advanced 
development  verification  testing. 

DT  during  systems  integration  is  most  often  conducted  at  the  contractor's  facility.  It  is  conducted  on 
components,  subsystems,  brassboard  configurations,  or  advanced  development  prototypes  to  evaluate 
the  potential  application  of  technology  and  related  design  approaches  before  system  demonstration. 
Component  interface  problems  and  equipment  performance  capabilities  are  evaluated.  The  use  of 
validated  M&S  tools  is  encouraged,  especially  during  the  early  phases  to  assess  those  areas  that,  for 
safety  or  testing  capability  limitations,  cannot  be  observed  directly  through  testing.  M&S  can  provide 
early  projections  of  systems  performance,  effectiveness,  and  suitability  and  can  reduce  testing  costs. 
Testing  may  include  initial  environmental  assessments. 

RAM  data  needs  to  be  collected  throughout  the  test  program.  Examples  of  reliability  development 
testing  (RDT)  processes  include  test,  analyze,  fix,  and  test  (TAFT)  and  test,  analyze,  and  fix  (TAAF),  which 
is  shown  in  Figure  6-1.  RAM  data  from  early  contractor  testing  provide  key  data  points,  including  an 
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estimate  of  initial  reliability,  and  become  part  of  the  systems  RAM  database  to  support  planning, 
projections,  and  tracking.  DT&E  conducted  during  the  SC&MPD  effort  of  EMD  provides  the  final 
technical  data  for  determining  a  systems  readiness  to  transition  into  LRIP.  It  is  conducted  using 
advanced  EDMs  and  is  characterized  by  engineering  and  scientific  approaches  under  controlled 
conditions.  During  this  same  time,  programs  on  OSD  OT&E  oversight  are  required  to  conduct  an  OA  in 
support  of  an  LRIP  decision.  The  DT&E  test  lead  should  consider  concurrent  DT/OT  during  this  period  to 
maximize  data  collection  and  minimize  test  resources  and  time.  The  qualification  testing  provides 
quantitative  and  qualitative  data  for  use  in  the  systems  evaluation.  The  evaluation  results  are  used  by 
the  development  community  and  are  also  provided  to  Service  and  OSD  decision  authorities.  These  tests 
measure  technical  performance  including  effectiveness,  RAM,  compatibility,  interoperability,  lA,  safety, 
and  supportability.  They  include  tests  of  human  engineering  and  technical  aspects  of  the  system. 
Demonstrations  indicating  that  engineering  is  reasonably  complete  and  that  solutions  to  all  significant 
design  problems  are  in  hand  are  also  included.  The  PM  should  use  DoD  4245. 7-M  (Reference  (ai))  to 
check  that  all  aspects  of  the  program  have  been  properly  accomplished  before  any  type  of  production 
begins. 

6.2.4  DT&E  During  P&D  Phase 

6.2.4.1  DT&E  During  LRIP 

Each  Service  has  different  and  specific  processes  incorporated  in  the  certification  for  lOT&E 
documentation.  For  example,  the  Navy  conducts  additional  DT&E  for  certification  called  technical 
evaluation.  This  is  a  DT&E  event  controlled  by  the  program  office  that  is  conducted  in  a  more 
operationally  realistic  test  environment.  The  Air  Force,  on  the  other  hand,  has  developed  a  guide  with  a 
structured  process  using  templates  to  assist  the  PM  in  assessing  the  programs  readiness  for  lOT&E. 

6.2.4.2  DT&E  After  FRPDR 

DT  may  be  necessary  after  the  FRP  decision  is  made.  This  testing  is  normally  tailored  to  verify  correction 
of  identified  design  problems  and  demonstrate  the  system  modifications  readiness  for  production.  This 
testing  is  conducted  under  controlled  conditions  and  provides  quantitative  and  qualitative  data.  It  is 
conducted  on  production  items  delivered  from  either  the  pilot  or  initial  production  runs.  To  ensure  that 
the  items  are  produced  according  to  contract  specification,  limited  quantity  production  sampling 
processes  are  used.  This  testing  determines  whether  the  system  has  successfully  transitioned  from 
engineering  development  prototype  to  production,  and  whether  it  meets  design  specifications. 

6.2.4.3  Production  Qualification  Test  (PQT) 

Qualification  testing  is  a  form  of  DT  that  verifies  the  design  and  manufacturing  process.  PQTs  are  formal 
contractual  tests  that  confirm  the  integrity  of  the  system  design  over  the  operational  and  environmental 
range  in  the  specification.  These  tests  usually  use  production  hardware  fabricated  to  the  proposed 
production  design  specifications  and  drawings.  Such  tests  typically  include  contractual  reliability  and 
maintainability  (R&M)  demonstration  tests  required  before  production  release.  PQT  must  be  completed 
before  FRP  in  accordance  with  Reference  (d) . 
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DCMA  typically  conducts  PQT  at  the  direction  of  the  program  office.  PQTs  may  be  conducted  on  LRIP 
items  to  ensure  the  maturity  of  the  manufacturing  process,  equipment,  and  procedures.  These  tests  are 
conducted  on  each  item  or  a  sample  lot  taken  at  random  from  the  first  production  lot  and  are  repeated 
if  the  process  or  design  is  changed  significantly  or  a  second  or  alternative  source  is  brought  online. 

These  tests  are  also  conducted  against  contractual  design  and  performance  requirements. 

6.2.5  DT&E  During  O&S  Phase 

The  DT&E  that  continues  after  IOC  or  initial  deployment  assesses  the  deployed  systems  operational 
readiness  and  supportability.  It  ensures  that  all  deficiencies  noted  during  previous  testing  have  been 
corrected,  evaluates  proposed  Pis  and  block  upgrades,  and  ensures  that  ILS  is  complete.  It  also  evaluates 
the  resources  on  hand  and  determines  whether  the  plans  to  ensure  operational  phase  readiness  and 
support  objectives  are  sufficient  to  maintain  the  system  for  the  remainder  of  its  acquisition  life  cycle. 

For  mature  systems,  DT&E  is  performed  to  assist  in  modifying  the  system  to  help  meet  new  threats,  add 
new  technologies,  aid  in  extending  service  life,  or  determine  the  effects  of  storage  on  system 
components.  The  individual  in  the  PMO  responsible  for  managing  the  package  of  support  functions 
required  to  field  and  maintain  the  readiness  and  operational  capability  of  major  weapon  systems, 
subsystems,  and  components  is  the  product  support  manager  (PSM).  The  PSM  is  responsible  for  all 
functions  related  to  weapon  system  readiness  in  support  of  the  PMs  life  cycle  management 
responsibilities.  The  PSM  should  provide  the  DT&E  community  with  the  best  insight  into  end-of-life 
issues. 

Once  a  system  approaches  the  end  of  its  usefulness,  the  testing  conducted  is  concerned  with  the 
monitoring  of  a  systems  current  state  of  operational  effectiveness,  suitability,  and  readiness  to 
determine  whether  major  upgrades  are  necessary  or  deficiencies  warrant  consideration  of  a  new  system 
replacement.  Tests  are  normally  conducted  by  the  OT  community. 

6.3  DT&E  Responsibilities 

DT&E  organizations  continue  to  monitor  and  observe  test  progress,  supporting  test  plan  development 
and  approval,  and  reporting  as  required.  During  the  EMD  phase,  the  OT&E  and  DT&E  organizations  must 
collaborate  in  conducting  OAs  and  other  forms  of  integrated  testing.  In  some  Services,  there  are  also 
independent  evaluation  organizations  that  assist  the  testing  organization  in  designing  and  evaluating 
DTs.  As  the  figure  shows,  system  DT  is  performed  principally  by  contractors  during  the  early 
development  stages  of  the  acquisition  cycle  and  by  Government  test/evaluation  organizations  during 
the  later  phases. 

6.3.1  Typical  Contractor  Testing  Responsibilities 

Contractor  testing  plays  a  primary  role  in  the  total  test  program,  and  the  results  of  contractor  tests  are 
useful  to  the  Government  evaluator  in  supporting  Government  test  objectives.  It  is  important  that 
Government  evaluators  oversee  contractor  system  tests  and  use  test  data  from  them  to  address 
Government  testing  issues.  It  is  not  uncommon  for  contractor  testing  to  be  conducted  at  Government 
test  facilities  because  contractors  often  do  not  have  the  required  specialized  facilities  (e.g.,  for  testing 
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hazardous  components  or  for  missile  flight  tests).  This  arrangement  enables  Government  evaluators  to 
monitor  the  tests  more  readily,  decreases  total  program  costs,  and  increases  Government  confidence  in 
the  test  results. 

The  PM  and  contracting  officer  must  capture  all  T&E  needs,  directions,  and  requirements  in  the  RFP 
using  Reference  (y) .  Normally,  an  RFP  requires  that  the  winning  contractor  submit  an  integrated  test 
concept  or  TEMP  within  a  short  period  after  contract  initiation  for  coordination  with  Government  test 
agencies  and  approval.  This  test  plan  should  include  all  tests  required  by  the  SOW,  specifications,  and 
testing  expected  as  part  of  the  contractor's  overall  engineering  development  and  integration  process. 

If  T&E  requirements  are  not  in  the  RFP,  then  chances  are  they  will  not  be  in  the  SOW  or  contractor's  bid. 
If  they  are  not  in  the  SOW,  then  the  Government  will  not  get  the  required  resources  and/or  higher  costs 
will  result.  If  the  contractor  has  misinterpreted  the  RFP  requirements  or  the  Government  has  not 
adequately  articulated  needed  testing  as  part  of  the  RFP,  then  the  iterative  process  of  amending  the 
contractor's  test  program  begins.  This  iterative  process  must  be  accomplished  within  limited  bounds  so 
the  contractor  can  meet  the  test  objectives  without  significant  effects  on  contract  cost,  schedule,  or 
scope. 

6.3.2  Typical  Government  Testing  Responsibilities 

Government  testing  builds  on  contractor  testing  and  is  performed  to  verify  and  demonstrate  how  well 
the  system  under  development  meets  its  technical  compliance  requirements,  to  provide  data  to  assess 
developmental  risk  for  decision  making,  and  to  ensure  that  the  technical  and  support  problems 
identified  in  previous  testing  have  been  corrected  and  that  all  critical  issues  to  be  resolved  by  testing 
have  been  adequately  considered.  All  previous  testing,  from  the  contractor's  bench  testing  through 
development  agency  testing  of  representative  prototypes,  is  considered  during  Government  evaluation. 

PMs  and  the  PMO  must  ensure  that  the  DASD(DT&E),  the  DOT&E,  and  their  designated  representatives 
have  full  and  immediate  access  to  all  records,  all  reports,  and  all  data,  including  but  not  limited  to  data 
from  all  tests,  system  logs,  execution  logs,  test  director  notes,  user/operator  assessments,  etc.  All  data 
include  but  are  not  limited  to  classified,  unclassified,  and  competition-sensitive  or  proprietary  data 
when  available.  Data  may  be  preliminary  and  must  be  identified  as  such. 

Government  materiel  development  organizations  include  major  materiel  acquisition  commands  and,  in 
some  cases,  operational  commands.  The  materiel  acquisition  commands  have  T&E  organizations  that 
conduct  Government  DT.  In  addition  to  monitoring  and  participating  in  contractor  testing,  these 
organizations  conduct  DT  on  selected  high-concern  areas  to  evaluate  the  adequacy  of  SE,  design, 
development,  and  performance  to  specifications.  The  PMO  must  be  involved  in  all  stages  of  testing  that 
these  organizations  perform. 

In  turn,  the  materiel  development/T&E  agencies  conduct  DT&E  of  the  systems  in  the  development  stage 
to  verify  that  they  meet  technical  specifications  and  have  the  potential  to  meet  operational 
requirements.  These  organizations  operate  Government  proving  grounds,  test  facilities,  and  labs.  They 
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must  be  responsive  to  the  needs  of  the  PM  by  providing  test  facilities,  personnel,  and  data  acquisition 
services,  as  required. 

6.4  Test  Program  Integration 

During  the  development  of  a  weapon  system,  many  tests  are  conducted  by  subcontractors,  the  prime 
contractor,  and  the  Government.  To  ensure  that  these  tests  are  properly  timed  and  adequate  resources 
are  available  and  to  minimize  unnecessary  testing,  a  coordinated  test  program  must  be  developed  and 
followed.  The  TEMP  normally  does  not  provide  a  sufficient  level  of  detail  concerning  contractor  or 
subcontractor  tests.  A  contractor  or  PMO  test  plan  must  also  be  developed  to  describe  these  lower  level 
tests  in  detail.  The  PM  is  responsible  for  coordinating  the  total  T&E  program.  The  PM  and/or  designated 
lead  (typically  the  Chief  Developmental  Tester  in  many  PMOs)  performs  this  task  with  the  assistance  of  a 
T&E  WIPT  whose  members  should  be  assembled  from  the  development  agency,  user,  technical  and 
operational  T&E,  logistics,  and  training  organizations.  The  PMO  must  remain  active  in  all  aspects  of 
testing  including  planning,  funding,  resourcing,  execution,  and  reporting.  The  PM  plays  an  important 
role  as  the  interface  between  the  contractor  and  the  Government  testing  community.  Recent  emphasis 
on  early  T&E  highlights  a  need  for  early  Government  tester  involvement  in  contractor  testing  and  early 
assessments  of  system  functions  in  the  mission  context  of  the  operational  environment  in  which  the 
system  will  be  used. 

The  PM  shall  identify  each  test  phase  or  major  event  in  the  TEMP  as  a  contractor  or  Government  DT&E. 
All  programs  must  plan  for  the  conduct  of  DT&E  or  integrated  test  that  includes  or  is  led  by  Government 
personnel,  so  as  to  provide  confidence  that  the  system  design  solution  is  on  track  to  execute  the 
operational  scenarios  identified  by  the  OTAs  in  accordance  with  Reference  (d) . 

Whenever  feasible,  DT&E  and  OT&E  events  should  be  combined,  if  doing  so  supports  technical  and 
operational  test  objectives,  to  gain  the  optimum  amount  of  testing  benefit  for  reasonable  cost  and  time. 
The  user  community  should  be  involved  early  in  test  planning  to  ensure  that  the  statement  of  desired 
capabilities  is  interpreted  correctly  and  tested  realistically.  Certain  events  can  be  organized  to  provide 
information  useful  to  developmental  and  operational  evaluators  and  lend  themselves  to  the  combined 
DT  and  OT  approach.  The  concept  is  to  conduct  a  single,  combined  test  program  that  produces  credible 
qualitative  and  quantitative  information  that  can  be  used  to  address  developmental  and  operational 
issues.  See  Chapter  8  of  this  Guide  for  additional  information. 

6.5  Areas  Of  DT&E  Focus 

The  products  of  DT&E  are  critical  to  the  acquisition  process.  DT&E  is  focused  on  identifying  acquisition 
risk;  improving  operational  insight  into  the  system;  providing  an  independent  risk  assessment;  providing 
characterization  of  system  functionality;  ensuring  document  specification  and  contract  compliance; 
assessing  system  readiness  for  lOT&E;  identifying  system  capabilities,  limitations,  and  deficiencies; 
providing  information  for  cost,  performance,  and  schedule  trade-offs;  using  integrated  testing  to 
produce  credible  qualitative  and  quantitative  data;  and  enabling  more  rapid  fielding.. 
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6.5.1  System  Life  Cycle  Testing 

Life  cycle  testing  is  performed  to  assess  the  effects  of  long-term  exposure  to  various  portions  of  the 
anticipated  environment.  These  tests  are  used  to  ensure  that  the  system  will  not  fail  prematurely 
because  of  metal  fatigue,  component  aging,  corrosion,  or  other  problems  caused  by  long-term  exposure 
to  environmental  effects.  Requirements  for  life  cycle  testing  must  be  identified  early  and  integrated  into 
system  TEMPs  and  test  plans,  and  testing  must  be  started  as  soon  as  practical  after  stable  prototypes 
are  available.  System  life  cycle  tests  can  be  time-consuming  and  costly;  therefore,  the  testing 
requirements  and  life  cycle  characteristics  must  be  carefully  analyzed  during  the  initial  test  design. 
Accelerated  life  cycle  testing  techniques  are  available  and  should  be  used  whenever  applicable. 
Collection  and  analysis  of  test  data  should  occur  throughout  the  testing  cycle.  If  life  cycle  characteristics 
are  ignored  until  the  end  of  the  testing  phase,  extensive  redesign  and  project  delays  may  result. 

6.5.2  Design  Evaluation  and  Verification  Testing 

Design  evaluation  and  verification  testing  are  conducted  by  the  contractor  and/or  the  development 
agency  with  the  primary  objective  of  influencing  system  design.  Design  evaluation  is  fully  integrated  into 
the  DT  cycle  in  order  to: 

Determine  whether  critical  system  technical  characteristics  are  achievable. 

Provide  data  for  refining  and  making  the  hardware  more  rugged  to  comply  with  technical  specification 
requirements. 

Eliminate  as  many  technical  and  design  risks  as  possible  or  to  determine  the  extent  to  which  they  are 
manageable. 

Provide  for  evolution  of  design  and  verification  of  the  adequacy  of  design  changes. 

Provide  information  in  support  of  development  efforts. 

Ensure  that  components,  subsystems,  and  systems  are  adequately  developed  before  OTs. 

6.5.3  Design  Limit  Testing  (DLT) 

DLTs  are  integrated  into  the  test  program  to  ensure  that  the  system  will  provide  adequate  performance 
when  operated  at  outer  performance  limits  and  when  exposed  to  environmental  conditions  expected  at 
the  extremes  of  the  operating  envelope.  The  tests  are  based  on  mission  profile  data.  Care  must  be  taken 
to  ensure  that  all  systems  and  subsystems  are  exposed  to  the  worst-case  environments,  with 
adjustments  made  because  of  stress  amplification  factors  and  cooling  problems;  for  example,  an  aircraft 
component  may  have  to  be  tested  at  temperature  extremes  from  an  arctic  environment  to  a  desert 
environment. 
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6.5.4  Testing  for  Reliability 

The  RAM  requirements  are  assessed  during  all  contractor  and  Government  testing.  The  reliability 
growth  plan  is  an  essential  indicator  of  how  RAM  will  be  assessed  and  reported  as  DT&E  progresses 
throughout  the  acquisition  and  testing  cycles.  Data  are  collected  from  each  test  event  and  placed  in  a 
RAM  database,  which  is  managed  by  the  development  agency.  Contractor  and  Government  DT  and  OT 
provide  a  measure  of  the  systems  common  RAM  performance  against  stated  specifications  in  a 
controlled  environment.  RAM  data  collection  during  DT  emphasizes  reliability  growth  and  comparison 
against  a  planned  reliability  growth  curve  (RGC)  that  should  ultimately  reach  the  specified  requirement. 
Early  projections  of  RAM  are  important  to  logistics  support  planners.  The  test  data  facilitate 
determination  of  spares  quantities,  maintenance  procedures,  and  support  equipment. 

Testing  for  reliability  is  a  planned  process  in  which  items  are  tested  under  actual  or  simulated  mission- 
profile  environments  to  disclose  design  deficiencies  and  to  provide  engineering  information  on  failure 
modes  and  mechanisms.  Testing  for  reliability  provides  a  basis  for  early  incorporation  of  corrective 
actions  and  verification  of  their  effectiveness  in  improving  the  reliability  of  equipment.  Testing  is 
conducted  under  controlled  conditions  with  simulated  operational  mission  and  environmental  profiles 
to  determine  design  and  manufacturing  process  weaknesses.  The  reliability  growth  testing  (RGT)  process 
emphasizes  reliability  growth  rather  than  a  numerical  measurement.  Reliability  growth  during  testing  is 
the  result  of  an  iterative  design  process  because  as  the  failures  occur,  the  problems  are  identified, 
solutions  are  proposed,  the  redesign  is  accomplished,  and  the  testing  for  reliability  continues.  The  TEMP 
requires  discussion  of  RGT  and  its  planned  use  on  a  program  (  Reference  (o) ). 

6.5.5  Readiness  to  Enter  lOT&E 

Throughout  the  EMD  phase,  DT&E  activities  should  be  pulling  together  assessments  of  technical 
performance  with  the  operational  context  in  which  the  system  will  be  operated.  Following  MS  C,  the 
majority  of  the  technical  testing  is  complete  and  the  DT  to  OT  transition  reports  should  be  developed  in 
preparation  for  lOT&E.  Each  Service  is  charged  with  developing  and  managing  its  own  operational  test 
readiness  review  (OTRR)  process;  in  addition,  all  programs  on  DASD(DT&E)  oversight  are  subject  to  an 
independent  AOTR  conducted  by  the  DASD(DT&E).  The  goal  of  the  AOTR  is  to  provide  the  CAE  with  an 
independent  assessment  of  the  risk  associated  with  the  system's  ability  to  meet  operational  suitability 
and  effectiveness  goals,  identify  system  and  subsystem  maturity  levels,  assess  programmatic  and 
technical  risk,  and  provide  risk  mitigation  recommendations. 

6.6  System  Design  For  Testing 

Built-in  test  (BIT),  built-in  test  equipment  (BITE),  and  automated  test  equipment  (ATE)  are  major  areas 
that  must  be  considered  from  the  start  of  the  design  effort.  During  PDR/CDR,  design  for  testability  is 
evaluated  as  a  condition  for  approval.  Design  for  testability  is  an  important  consideration  that  addresses 
the  need  to: 

Collect  data  during  the  development  process  concerning  particular  performance  characteristics. 
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Enable  efficient  and  economical  production  by  providing  ready  access  to  and  measurement  of 
appropriate  acceptance  parameters. 

Enable  rapid  and  accurate  assessment  of  the  status  of  the  product  to  the  lowest  repairable  element 
when  deployed. 

Many  hardware  systems  have  testing  circuits  designed  and  built  in.  This  early  planning  by  design 
engineers  allows  easy  testing  for  fault  isolation  of  circuits,  both  in  system  development  phases  and 
during  OT.  This  type  of  circuit  design  requires  early  planning  by  the  PM  to  ensure  that  the  RFP  includes 
the  requirement  for  designed/BIT  capability.  Evaluation  of  these  BIT/BITE/ATE  systems  must  be  included 
in  the  test  program. 

6.7  DT&E  Of  Limited  Procurement  Quantity  Programs 

Programs  that  involve  the  procurement  of  relatively  few  items,  such  as  satellites,  some  large  missiles, 
ships,  and  unique  intelligence  equipment,  typically  over  an  extended  period,  are  normally  subjected  to 
modified  DT&E.  Occasionally,  a  unique  test  approach  that  deviates  from  the  standard  timing  and 
reporting  schedule  will  be  used.  The  DT&E  principle  of  iterative  testing  starting  with  components, 
subsystems,  prototypes,  and  first-production  models  of  the  system  is  normally  applied  to  limited 
procurements.  It  is  important  that  DT&E  and  OT&E  organizations  work  together  to  ensure  that 
integrated  T&E  plans  are  adapted/tailored  to  the  overall  acquisition  strategy. 

6.8  Summary 

DT&E  is  a  critical  part  of  an  iterative  material  acquisition  process  of  designing,  building,  testing, 
identifying  deficiencies,  fixing,  retesting,  and  repeating.  It  is  performed  in  the  factory,  in  the  laboratory, 
and  on  the  proving  ground  by  contractors  and  the  Government.  Contractor  and  Government  testing  is 
combined  into  one  integrated  test  program  and  conducted  to  determine  whether  the  performance 
requirements  have  been  met  and  to  provide  data  to  the  decision  authority. 
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Test  &  Evaluation  Management  Guide 

Chapter  7  -  DT&E  Support  of  Technical  Reviews  and  Milestone  Decision 

7.1  Introduction 

Throughout  the  acquisition  process,  DT&E  is  oriented  around  the  demonstration  of  technical  compliance 
to  specifications  showing  the  completeness  and  adequacy  of  SE,  design,  development,  and  performance. 
DT&E  is  a  tool  that  shows  that  the  system  performs  as  specified,  or  having  identified  deficiencies,  shows 
that  those  deficiencies  have  been  corrected  and  the  system  is  ready  for  OT.  DT&E  process  results  are 
used  throughout  SE  efforts  to  provide  valuable  data  in  support  of  formal  technical  reviews.  DT&E 
inherently  supports  AoA  updates,  design  reviews,  requirements  updates,  and  readiness  for  OT  and  use 
by  operational  units,  as  outlined  in  descriptions  of  the  technical  reviews  detailed  within  this  chapter. 

A  pictorial  roadmap  of  key  activities  in  the  systems  acquisition  processes-the  Integrated  Defense 
Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System  chart-is  available  at 
https://ilc.dau.mil .  The  chart  is  a  training  aid  for  DAD  courses  and  illustrates  the  interaction  of  the  three 
key  processes  that  must  work  in  concert  to  deliver  the  capabilities  required  by  the  Warfighters:  the 
requirements  process  (JCIDS  process);  the  acquisition  process  (Defense  Acquisition  System  process);  and 
the  program  and  budget  development  process  (PPBE  process). 

This  chapter  describes  DT&Es  support  to  the  technical  reviews  essential  to  the  SE  effort.  As  depicted  in 
Figure  7-1,  all  phases  of  the  acquisition  life  cycle  employ  DT&E  to  facilitate  various  developmental 
evaluation  requirements.  The  evaluations  span  from  early  prototype  and  system  alternative 
assessments  through  product  development  and  verification  of  system  specifications  and  PQTs. 

7.2  DT&E  and  The  Review  Process 
7.2.1  Technical  Reviews 

Technical  reviews  and  audits  are  conducted  as  part  of  the  overall  SE  process  to  ensure  that  the  design 
meets  the  system,  subsystem,  and  software  specifications  and  to  gain  insight  into  development  progress 
and  risks.  Each  review  or  audit  is  unique  in  its  timing,  orientation,  and  exit/entrance  criteria. 

The  ICD  is  very  important  in  this  process.  It  is  the  top-level  document  used  as  a  benchmark  to  compare 
contractor  progress  in  designing  and  developing  the  desired  product.  Criteria  and  timing  for  these 
formal  technical  reviews  and  audits  can  be  found  in  Chapter  4  (Systems  Engineering)  of  Reference  (i). 
Detailed  official  checklists  for  all  DoD  technical  reviews  and  audits  can  be  found  at  the  SE  CoP,  available 
via  the  DAU  Acquisition  Community  Connection  at  https://acc.dau.mil/CommunityBrowser.aspx  . 
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Figure  7-1  Example  DT&E  Activities  Across  Phases  of  the  Acquisition  Life  Cycle 


7.2.2  Testing  in  Support  of  Technical  Reviews 

The  testing  community  must  be  continually  involved  in  supporting  technical  reviews  at  the  subsystem 
and  system  levels.  The  timing  of  these  events,  illustrated  in  Figure  7-2,  will  vary  somewhat  from 
program  to  program.  Lower  level  design  reviews  (typically  at  subsystem  level)  will  lead  to  system-level 
reviews.  Decisions  made  at  these  reviews  have  major  impacts  on  the  system  test  design  and  the 
resources  required  to  test  and  may  impact  the  TEMP  and  other  documentation.  Technical  reviews  and 
their  proposed  timing  on  a  given  program  are  typically  discussed  in  the  programs  SEP.  Figure  7-3 
illustrates  the  program  specifications,  which  are  key  aspects  of  the  technical  review  process,  and  how 
they  are  developed  and  matured  across  the  system  life  cycle. 

DT&E  is  a  process  to  support  the  developmental  evaluation  of  a  system  and  evaluate  the  maturation  of 
KPPs,  KSAs,  TPMs  and  CTPs,  and  parameters.  In  addition,  technical  performance  measures  provide  a 
technique  for  predicting  the  future  value  of  a  key  technical  performance  parameter  of  the  higher  level 
product  under  development,  based  on  current  assessments  of  products  lower  in  the  system  structure. 
During  the  technical  review  process,  these  parameters  and  measures  assist  in  the  evaluation  of  system 
development. 
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Figure  7-2  An  Example  of  Technical  Review  Timing  Across  the  Acquisition  Life  Cycle 
7.2.3  Technical  Design  Reviews  and  Audits  Across  the  Life  Cycle 
7.2. 3.1  Summary  of  MSA  Phase  Reviews 

Initial  Technical  Review  (ITR):  The  ITR  is  a  multi-disciplined  review  held  to  ensure  that  a  program's 
technical  baseline  is  sufficiently  rigorous  to  support  a  valid  cost  estimate.  The  ITR  assesses  the  capability 
needs  and  materiel  solution  approach  of  a  proposed  program  and  verifies  that  the  requisite  research, 
development,  test  and  evaluation,  engineering,  logistics,  and  programmatic  bases  for  the  program 
reflect  the  complete  spectrum  of  technical  challenges  and  risks.  Additionally,  the  ITR  ensures  the 
historical  and  prospective  drivers  of  system  life-cycle  cost  have  been  quantified  to  the  maximum  extent 
and  that  the  range  of  uncertainty  in  these  parameters  has  been  captured  and  reflected  in  the  program 
cost  estimates.  The  ITR  is  a  multi-disciplined  review  held  to  ensure  that  a  programs  technical  baseline  is 
sufficiently  rigorous  to  support  a  valid  cost  estimate.  The  ITR  assesses  the  capability  needs  and  materiel 
solution  approach  of  a  proposed  program  and  verifies  that  the  requisite  research,  development,  T&E, 
engineering,  logistics,  and  programmatic  bases  for  the  program  reflect  the  complete  spectrum  of 
technical  challenges  and  risks.  Additionally,  the  ITR  ensures  that  the  historical  and  prospective  drivers  of 
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system  LCC  have  been  quantified  to  the  maximum  extent  and  that  the  range  of  uncertainty  in  these 
parameters  has  been  captured  and  reflected  in  the  program  cost  estimates. 
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Figure  7-3  Specifications  Summary 


Alternative  Systems  Review  (ASR):  The  ASR  is  a  multi-disciplined  technical  review  to  ensure  that  the 
technical  baseline  agrees  with  the  customer's  needs  and  expectations  and  that  the  system  under  review 
can  proceed  into  the  TD  phase.  The  ASR  should  be  completed  prior  to  and  provide  information  for  MS  A. 
Generally,  the  ASR  assesses  the  preliminary  materiel  solutions  that  have  been  evaluated  during  the  MSA 
phase  and  ensures  that  the  one  or  more  proposed  materiel  solutions  have  the  best  potential  to  be  cost- 
effective,  affordable,  operationally  effective,  and  suitable  and  can  be  developed  to  provide  a  timely 
solution  to  a  need  at  an  acceptable  level  of  risk.  Of  critical  importance  to  this  review  is  the 
understanding  of  available  system  concepts  to  meet  the  capabilities  described  in  the  ICD  and  to  meet 
the  affordability,  operational  effectiveness,  technology  risk,  and  suitability  goals  inherent  in  each 
alternative  concept. 

Acquisition  policy  requires  prototyping  in  the  TD  phase;  therefore,  the  ASR  should  identify  key  system 
elements  that  two  or  more  competing  teams  will  prototype  prior  to  MS  B. 
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7. 2. 3. 2  Summary  of  TD  Phase  Reviews 

Integrated  Baseline  Review  (IBR):  Baselines  formally  document  a  product  at  some  given  level  of  design 
definition.  They  are  references  for  the  subsequent  development  to  follow.  Most  DoD  systems  are 
developed  using  the  three  classic  baselines:  functional,  allocated,  and  product.  Though  the  program- 
unique  specifications  are  the  dominant  baseline  documentation,  they  alone  do  not  constitute  a  baseline. 
Additional  documents  include  both  end  and  enabling  product  descriptions.  End-product  baseline 
documents  normally  include  those  describing  system  requirements,  functional  architecture,  physical 
architecture,  technical  drawing  package,  and  requirements  traceability.  APBs  are  high-level  assessments 
of  program  maturity  and  viability.  Configuration  baselines  are  system  descriptions.  The  IBR  establishes 
the  Performance  Measurement  Baseline  (PMB)  for  Earned  Value  Management  (EVM).  PMs  may  use  the 
IBR  throughout  the  program  when  EVM  is  required. 

Prototype  Reviews:  The  definition,  development,  and  demonstration  of  prototypes  of  system, 
subsystem,  assembly,  and  component  may  require  the  application  of  SE  technical  management 
processes  and  technical  processes  and  appropriate  prototype  technical  reviews.  Technical  reviews  in 
support  of  competitive  prototyping  and  SE  technical  reviews  such  as  prototype  functional  review, 
prototype  PDR,  or  prototype  CDR  can  be  beneficial  to  support  the  technical  approach  and  program 
plans.  Application,  use,  and  timing  of  prototype  reviews  are  at  the  discretion  of  the  program  office  and 
prototype  developer  and  are  outlined  in  the  programs  SEP. 

System  Requirements  Review  (SRR):  The  SRR  is  a  multi-disciplined  technical  review  to  ensure  that  the 
system  under  review  can  proceed  into  initial  systems  development  and  that  all  system  requirements 
and  performance  requirements  derived  from  the  ICD  or  draft  CDD  are  defined  and  testable  and  are 
consistent  with  cost,  schedule,  risk,  technology  readiness,  and  other  system  constraints.  Generally,  this 
review  assesses  the  system  requirements  as  captured  in  the  system  specification  and  ensures  that  the 
system  requirements  are  consistent  with  the  approved  materiel  solution  (including  its  support  concept) 
as  well  as  available  technologies  resulting  from  the  prototyping  effort. 

The  SRR  is  conducted  to  ascertain  progress  in  defining  system  technical  requirements.  The  results  of 
integrated  test  planning  are  also  reviewed  to  ensure  the  adequacy  of  planning  to  assess  the  design  and 
to  identify  risks.  Issues  of  testability  of  requirements  should  be  discussed.  The  SRR  is  intended  to 
confirm  that  the  user's  operational  requirements  are  sufficiently  well  understood  and  have  been 
translated  into  system-specific  technical  requirements  to  permit  the  developer  (contractor)  to  establish 
an  initial  system-level  requirements  baseline.  The  SRR  determines  the  direction  and  progress  of  the  SE 
effort  and  the  degree  of  convergence  upon  a  balanced  and  complete  configuration  baseline.  It  is 
normally  held  during  the  TD  phase,  but  it  may  be  repeated  after  the  start  of  EMD  phase  to  clarify  the 
contractor's  understanding  of  redefined  or  new  user  requirements. 

System  Functional  Review  (SFR):  The  SFR  is  a  multi-disciplined  technical  review  to  ensure  that  the 
systems  functional  baseline  is  established  and  has  a  reasonable  expectation  of  satisfying  the 
requirements  of  the  ICD  or  draft  CDD  within  the  currently  allocated  budget  and  schedule.  It  completes 
the  process  of  defining  the  items  or  elements  below  system  level.  This  review  assesses  the 
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decomposition  of  the  system  specification  to  system  functional  specifications,  ideally  derived  from  use- 
case  analysis.  A  critical  component  of  this  review  is  the  development  of  representative  operational  use 
cases  for  the  system.  System  performance  and  the  anticipated  functional  requirements  for  operations 
maintenance  and  sustainment  are  assigned  to  subsystems,  hardware,  software,  or  support  after 
detailed  analysis  of  the  architecture  and  the  environment  in  which  it  will  be  employed.  The  SFR 
determines  whether  the  systems  functional  definition  is  fully  decomposed  to  its  lower  level  and  whether 
IPTs  are  prepared  to  start  preliminary  design. 

As  part  of  the  SFR,  the  systems  lower  level  performance  requirements  are  evaluated  to  determine 
whether  they  are  fully  defined  and  consistent  with  the  system  concept,  and  whether  traceability  of 
lower  level  systems  requirements  to  top-level  system  performance  and  the  CDD  is  maintained.  The  SFR 
results  in  two  major  products:  the  final  version  of  the  system  performance  specification  and  draft 
version  of  the  performance  specifications,  which  describe  the  items  below  system  level  (item 
performance  specifications).  The  SFR  is  the  first  review  that  begins  to  allocate  requirements  to 
separated  subsystems.  As  such,  it  is  also  the  first  review  in  which  the  need  for  ICDs  becomes  necessary 
to  define  areas  of  responsibility  and  constraints  requiring  coordination.  A  successful  review  is  predicated 
on  the  IPTs  determination  that  the  system  performance  requirements,  lower  level  performance 
requirements,  and  plans  for  design  and  development  form  a  satisfactory  basis  for  proceeding  into 
preliminary  design. 

Software  Specification  Review  (SSR):  The  SSR  is  a  critical  subsystem  review,  unique  to  software¬ 
intensive  systems  and  held  prior  to  the  subsystem  PDR.  The  SSR  is  a  formal  review  of  the  computer 
software  configuration  item  (CSCI)  requirements,  normally  held  after  an  SFR  but  before  the  start  of  a 
CSCI  preliminary  design.  Its  purpose  is  to  validate  the  allocated  baseline  for  preliminary  CSCI  design, 
ensure  traceability  of  software  requirements  to  higher  level  requirements,  and  establish  software  test 
criteria.  See  also  Chapter  16  of  this  Guide. 

Preliminary  Design  Review  (PDR):  The  PDR  is  a  technical  assessment  establishing  the  systems  allocated 
baseline  to  ensure  that  the  system  under  review  has  a  reasonable  expectation  of  being  judged 
operationally  effective  and  suitable.  This  review  assesses  the  allocated  design  documented  in  subsystem 
product  specifications  for  each  Cl  in  the  system  and  ensures  that  each  function  in  the  functional  baseline 
has  been  allocated  to  one  or  more  system  CIs.  The  PDR  establishes  the  system-level  allocated  baseline 
(hardware,  software,  human/support  systems)  and  underlying  architectures  to  ensure  that  the  system 
under  review  has  a  reasonable  expectation  of  satisfying  the  requirements  within  the  currently  allocated 
budget  and  schedule. 

For  complex  systems,  a  PDR  may  be  conducted  incrementally  at  the  subsystem  level  for  each  Cl.  These 
incremental  reviews  lead  to  an  overall  system-level  PDR.  The  PDR  evaluates  the  set  of  subsystem 
requirements  to  determine  whether  they  correctly  and  completely  implement  all  system  requirements 
allocated  to  the  subsystem.  The  PDR  also  determines  whether  subsystem  requirements  trace  with  the 
system  design.  A  successful  PDR  is  predicated  on  the  determination  that  the  subsystem  requirements, 
subsystem  preliminary  design,  results  of  peer  reviews,  and  plans  for  development,  testing,  and 
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evaluation  form  a  satisfactory  basis  for  proceeding  into  detailed  design  and  test  procedure 
development. 

7. 2.3. 3  Summary  of  EMD  Phase  Reviews 

Critical  Design  Review  (CDR):  The  CDR  is  a  key  point  within  the  EMD  phase.  The  CDR  is  a  multi- 
disciplined  technical  review  establishing  the  initial  product  baseline  to  ensure  that  the  system  under 
review  has  a  reasonable  expectation  of  satisfying  the  requirements  of  the  CDD  within  the  currently 
allocated  budget  and  schedule.  Incremental  CDRs  are  held  for  each  Cl  culminating  with  a  system-level 
CDR.  This  review  assesses  the  final  design  as  captured  in  product  specifications  for  each  Cl  in  the  system 
and  ensures  that  each  product  specification  has  been  captured  in  detailed  design  documentation.  CIs 
may  consist  of  hardware  and  software  elements  and  include  items  such  as  airframe/hull,  avionics, 
weapons,  crew  systems,  engines,  trainers/training,  support  equipment,  etc.  Product  specifications  for 
hardware  enable  the  fabrication  of  CIs  and  include  production  drawings.  Product  specifications  for 
software  enable  coding  of  the  CSCI.  The  CDR  evaluates  the  proposed  baseline  (Build  To  documentation) 
to  determine  whether  the  system  design  documentation  (initial  product  baseline,  including  item  detail 
specifications,  material  specifications,  process  specifications)  is  satisfactory  to  start  initial 
manufacturing. 

The  CDR  brings  closure  to  technical  risk  mitigation  and  alternate  design  paths  in  detailed  system  design. 
Once  the  product  baseline  is  established,  opportunities  to  improve  performance  or  reduce  LCCs  are 
severely  limited.  Changes  to  support  equipment,  training  requirements,  logistics  and  supply  elements, 
interoperability,  and  performance  can  only  be  accomplished  through  a  formal  engineering  change 
proposal.  All  technical  risk  should  be  reduced  to  acceptable  levels,  and  remaining  program  execution 
risk  resulting  from  resource  or  schedule  shortfalls  should  be  addressed  quickly  or  it  will  jeopardize 
program  success. 

For  complex  systems,  a  CDR  may  be  conducted  for  each  subsystem  and  logistics  element.  These 
incremental  reviews  lead  to  an  overall  system  CDR.  Incremental  design  reviews  are  usually  defined  at 
ICD  boundaries.  When  incremental  reviews  have  been  conducted,  additional  risk  is  introduced  until  the 
overall  system  CDR  establishes  the  complete  system  product  baseline.  Each  incremental  CDR  closes  a 
functional  or  physical  area  of  design  to  modification  regardless  of  when  it  is  held.  This  completed  area  of 
design  may  need  to  be  reopened  if  open  areas  cannot  achieve  desired  performance  in  isolation.  If  the 
schedule  is  being  preserved  through  parallel  design  and  build  decisions,  any  system  deficiencies  that 
lead  to  reopening  design  will  result  in  rework  and  possible  material  scrap. 

At  CDR,  the  EMD  process  results  in  a  detailed  product  baseline  for  the  system,  hardware,  software, 
support  equipment,  training  systems,  system  integration  laboratory,  and  technical  data.  The  subsystem 
detailed  designs  and  logistics  elements  are  evaluated  to  determine  whether  they  correctly  and 
completely  implement  all  allocated  system  requirements  and  whether  the  CDD  traceability  to  final 
system  detail  design  is  maintained.  The  overall  system-level  CDR  is  not  only  approval  of  the  system 
product  baseline  but  also  approval  of  the  product  baselines  for  maintainability,  supportability,  and 
logistics  elements.  A  successful  review  is  predicated  on  the  review  chairperson's  determination  that  the 
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subsystem  requirements,  subsystem  detail  design,  results  of  peer  reviews,  and  plans  for  T&E  form  a 
satisfactory  basis  for  proceeding  into  system  fabrication,  demonstration,  and  test. 

Test  Readiness  Review  (TRR):  The  TRR  is  a  multi-disciplined  technical  review  designed  to  ensure  that 
the  subsystem  or  system  under  review  is  ready  to  proceed  into  formal  test.  The  TRR  assesses  test 
objectives,  test  methods  and  procedures,  scope  of  tests,  and  safety  and  confirms  that  required  test 
resources  have  been  properly  identified  and  coordinated  to  support  planned  tests.  The  TRR  verifies  the 
traceability  of  planned  tests  to  program  requirements  and  user  needs.  It  determines  the  completeness 
of  test  procedures  and  their  compliance  with  test  plans  and  descriptions.  The  TRR  also  assesses  the 
system  under  review  for  development  maturity,  cost/  schedule  effectiveness,  and  risk  to  determine 
readiness  to  proceed  to  formal  testing. 

The  TRR  as  a  tool  can  be  used  to  support  all  tests  in  all  phases  of  an  acquisition  program,  including 
testing  within  a  system-of-systems  context.  The  PMs  and  the  T&E  WIPT  should  tailor  any  TRR  to  the 
specific  acquisition  phase,  the  specific  planned  tests,  and  the  identified  level  of  risk  within  the  program. 
The  scope  of  the  review  is  directly  related  to  the  risk  level  associated  with  performing  the  planned  tests 
and  the  importance  of  the  test  evaluation  results  to  overall  program  success.  The  scope  of  a  planned 
TRR  should  align  with  the  requirements  verification  matrix  in  the  programs  SEP. 

Readiness  to  convene  a  TRR  is  predicated  on  the  PMs  and  T&E  WIPTs  determination  that  preliminary, 
functional,  and  pre-qualification  test  evaluation  results  form  a  satisfactory  basis  for  proceeding  with  a 
TRR  and  subsequent  initiation  of  formal,  system-level  test.  The  TRR  should  answer  the  following 
questions: 

•  Why  are  we  testing?  What  is  the  purpose  of  the  planned  test?  Does  the  planned  test  verify  a 
requirement  that  is  directly  traceable  back  to  a  system  specification  or  other  program 
requirement? 

•  What  are  we  testing  (subsystem,  system,  system  of  systems,  other)?  Is  the  configuration  of  the 
system  under  test  sufficiently  mature,  defined,  and  representative  to  accomplish  planned  test 
objectives  and/or  support  defined  program  objectives? 

•  Are  we  ready  to  begin  testing?  Have  all  planned  preliminary,  informal,  functional,  unit-level, 
subsystem,  system,  and  qualification  tests  been  conducted,  and  are  the  results  satisfactory? 

•  What  is  the  expected  result  and  how  can/do  the  test  evaluation  results  affect  the  program? 

•  Is  the  planned  test  properly  resourced  (people,  test  article  or  articles,  facilities,  data  systems, 
support  equipment,  logistics,  etc.)? 

•  What  are  the  risks  associated  with  the  tests  and  how  are  they  being  mitigated? 

•  What  are  the  hazards  and  ESOH  risks  associated  with  the  specific  testing? 

•  Have  the  necessary  Safety  Releases  from  the  PM  been  provided  to  developmental  and 
operational  testers  prior  to  any  test  using  personnel? 

•  What  is  the  fail-back  plan  should  a  technical  issue  or  potential  showstopper  arise  during  testing? 

Flight  Readiness  Review  (FRR):  The  ERR  is  a  type  of  Test  Readiness  Review,  and  is  applicable  only  to 
aviation  programs.  It  assesses  the  readiness  to  initiate  and  conduct  flight  tests  or  flight  operations. 
Typically,  FRR  approval  requires  the  aviation  system  to  be  under  configuration  management,  a  flight 
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clearance  to  be  issued  by  the  technical  authority,  the  flight  test  plan(s)  to  be  approved,  and  discrepancy 
tracking  and  risk  assessment  processes  to  be  in  place. 

System  Verification  Review  (SVR):  The  SVR  is  a  multi-disciplined  product  and  process  assessment  to 
ensure  that  the  system  under  review  can  proceed  into  LRIP  and  FRP  within  cost  (program  budget), 
schedule  (program  schedule),  risk,  and  other  system  constraints.  Generally,  this  review  is  an  audit  trail 
from  the  SFR.  It  assesses  the  system  functionality  and  determines  whether  the  system  meets  the 
functional  requirements  documented  in  the  functional  baseline.  The  SVR  establishes  and  verifies  final 
product  performance.  The  SVR  is  often  conducted  concurrently  with  the  production  readiness  review 
(PRR).  The  FCA  may  also  be  conducted  concurrently  with  the  SVR. 

Functional  Configuration  Audit  (FCA):  The  FCA  is  the  formal  examination  of  the  as  tested  characteristics 
of  a  Cl  (hardware  and  software)  with  the  objective  of  verifying  that  actual  performance  complies  with 
design  and  interface  requirements  in  the  functional  baseline.  It  is  essentially  a  review  of  the  CIs 
test/analysis  data,  including  software  unit  test  results,  to  validate  that  the  intended  function  or 
performance  stated  in  its  specification  is  met.  For  the  overall  system,  this  would  be  the  system 
performance  specification.  For  large  systems,  FCAs  may  be  conducted  on  lower  level  CIs  for  specific 
functional  areas  and  address  non-adjudicated  discrepancies  as  part  of  the  system-level  FCA.  A  successful 
FCA  typically  demonstrates  that  the  EMD  product  is  sufficiently  mature  for  entrance  into  LRIP. 

During  the  FCA,  all  relevant  test  data  are  reviewed  to  verify  that  the  item  has  performed  as  required  by 
its  functional  and/or  allocated  configuration  identification.  The  audit  is  conducted  on  the  item 
representative  (prototype  or  production)  of  the  configuration  to  be  released  for  production.  The  audit 
consists  of  a  review  of  the  contractors  test  procedures  and  results.  The  information  provided  will  be 
used  during  the  FCA  to  determine  the  status  of  planned  tests. 

The  SVR  is  a  system-level  configuration  audit  that  may  be  conducted  after  system  testing  is  completed. 
The  objective  is  to  verify  that  the  actual  performance  of  the  Cl  (the  production  configuration),  as 
determined  through  testing,  complies  with  its  item  specifications  (performance)  and  to  document  the 
results  of  the  qualification  tests.  The  SVR  and  FCA  are  often  performed  at  the  same  time;  however,  if 
sufficient  test  results  are  not  available  at  the  FCA  to  ensure  that  the  Cl  will  perform  in  its  operational 
environment,  the  SVR  can  be  scheduled  for  a  later  time. 

Production  Readiness  Review  (PRR):  The  PRR  examines  a  program  to  determine  whether  the  design  is 
ready  for  production  and  whether  the  prime  contractor  and  major  subcontractors  have  accomplished 
adequate  production  planning  without  incurring  unacceptable  risks  that  will  breach  thresholds  of 
schedule,  performance,  cost,  or  other  established  criteria.  The  review  examines  risk;  it  determines 
whether  production  or  production  preparations  identify  unacceptable  risks  that  might  breach  thresholds 
of  schedule,  performance,  cost,  or  other  established  criteria.  The  review  evaluates  the  full,  production- 
configured  system  to  determine  whether  it  correctly  and  completely  implements  all  system 
requirements.  The  review  determines  whether  the  traceability  of  final  system  requirements  to  the  final 
production  system  is  maintained. 
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The  PRR  examines  the  readiness  of  the  manufacturing  processes,  the  quality  management  system,  and 
the  production  planning  (i.e.,  facilities,  tooling  and  test  equipment  capacity,  personnel  development  and 
certification,  process  documentation,  inventory  management,  supplier  management).  A  successful 
review  is  predicated  on  the  plans  for  design  and  development  addressing  the  system  performance 
requirements  and  lower  level  performance  requirements,  and  forms  a  satisfactory  basis  for  proceeding 
into  preliminary  design.  For  additional  information  see  Chapter  9  of  this  Guide. 

7. 2.3.4  Summary  of  P&D  Phase  Reviews 

Physical  Configuration  Audit  (PCA):  The  PCA  examines  the  actual  configuration  of  an  item  being 
produced.  It  verifies  that  the  related  design  documentation  matches  the  item  as  specified  in  the 
contract  and  as  built.  In  addition  to  the  standard  practice  of  ensuring  product  verification,  the  PCA 
confirms  that  the  manufacturing  processes,  quality  control  system,  measurement  and  test  equipment, 
and  training  are  adequately  planned,  tracked,  and  controlled.  The  PCA  validates  many  of  the  supporting 
processes  used  by  the  contractor  in  the  production  of  the  item  and  verifies  other  elements  of  the  item 
that  may  have  been  impacted/redesigned  after  completion  of  the  SVR. 

A  PCA  is  normally  conducted  when  the  Government  plans  to  control  the  detail  design  of  the  item  it  is 
acquiring  via  the  technical  data  package.  When  the  Government  does  not  plan  to  exercise  such  control 
or  purchase  the  items  technical  data  package  (e.g.,  performance  based  procurement),  the  contractor 
should  conduct  an  internal  PCA  to  define  the  starting  point  for  controlling  the  detail  design  of  the  item 
and  establishing  a  product  baseline.  The  PCA  is  complete  when  the  design  and  manufacturing 
documentation  matches  the  item  as  specified  in  the  contract  and  as  built. 

Assessment  of  Operational  Test  Readiness  (AOTR):  Prior  to  CAE  determination  of  readiness  for  lOT&E, 
the  DASD(DT&E)  conducts  an  independent  AOTR  for  all  ACAT  I  and  lA  programs  as  well  as  special 
interest  programs  designated  by  the  DASD(DT&E).  The  AOTR  focuses  on  the  technical  and  materiel 
readiness  of  the  program  to  proceed  into  lOT&E.  Assessment  results  are  based  on  capabilities 
demonstrated  in  DT&E  and  earlier  OAs.  A  DT&E  report  of  results  and  the  progress  assessment  must  be 
provided  to  the  DASD(DT&E)  and  DOT&E  prior  to  the  AOTR.  That  report  can  be  a  written  document  or  a 
briefing  to  DASD(DT&E)  and  DOT&E  representatives  and  should  include  the  following:  an  analysis  of  the 
systems  progress  in  achieving  CTPs;  satisfaction  of  approved  lOT&E  entrance  criteria;  a  technical  risk 
assessment;  level  of  software  maturity  and  status  of  software  trouble  reports;  and  predicted  lOT&E 
results,  including  the  impacts  of  any  shortcomings  on  the  systems  expected  performance  during  lOT&E. 
The  report  will  be  provided  prior  to  the  CAEs  determination  of  system  readiness.  This  will  allow  OSD 
time  to  formulate  and  provide  its  recommendation  to  the  CAE.  All  appropriate  developmental  and 
operational  T&E  organizations  should  be  invited  to  the  lOT&E  readiness  review. 

The  goal  of  the  AOTR  is  to  assess  the  risk  associated  with  the  system's  ability  to  meet  operational 
suitability  and  effectiveness  goals,  identify  system  and  subsystem  maturity  levels,  assess  programmatic 
and  technical  risk,  and  provide  risk  mitigation  recommendations.  The  results  of  the  AOTR  are  provided 
to  the  USD(AT&L),  DOT&E,  and  CAE.  The  CAE  must  consider  the  results  of  the  AOTR  before  making  a 
determination  of  materiel  readiness  for  lOT&E. 
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Operational  Test  Readiness  Review  (OTRR):  The  OTRR  is  a  multi-disciplined  product  and  process 
assessment  to  ensure  that  the  system  can  proceed  into  lOT&E  with  a  high  probability  of  success  and 
that  the  system  is  effective  and  suitable  for  service  introduction.  The  FRP  decision  may  hinge  on  this 
successful  determination.  The  understanding  of  available  system  performance  in  the  operational 
environment  to  meet  the  CPD  is  important  to  the  OTRR.  Consequently,  it  is  important  that  the  test 
addresses  and  verifies  system  reliability,  maintainability,  and  supportability  performance  and 
determines  whether  the  hazards  and  ESOH  residual  risks  are  manageable  within  the  planned  testing 
operations.  The  OTRR  is  complete  when  the  SAE  evaluates  and  determines  materiel  system  readiness 
for  lOT&E.  The  OTRR  risk  assessment  checklist  is  designed  as  a  technical  review  preparation  tool  and 
should  be  used  as  the  primary  guide  for  assessing  risk  during  the  review.  This  checklist  is  available  on  the 
SE  CoP  via  the  DAU  Acquisition  Community  Connection  at  https://acc.dau.mil/CommunityBrowser.aspx  . 

7. 2. 3. 5  Summary  of  (O&S)  Phase  Reviews 

In-Service  Review  (ISR):  The  ISR  is  a  multi-disciplined  product  and  process  assessment  to  ensure  that 
the  system  under  review  is  operationally  employed  with  well-understood  and  managed  risk.  This  review 
is  intended  to  characterize  the  in-service  health  of  the  deployed  system.  It  provides  an  assessment  of 
risk,  readiness,  technical  status,  and  trends  in  a  measurable  form.  These  assessments  substantiate  in- 
service  support  budget  priorities.  The  consistent  application  of  sound  programmatic,  SE,  and  logistics 
management  plans,  processes,  and  sub-tier  in-service  stakeholder  reviews  will  help  achieve  ISR 
objectives.  Example  support  groups  include  the  System  Safety  Working  Group  and  the  Integrated 
Logistics  Management  Team.  A  good  supporting  method  is  the  effective  use  of  available  Government 
and  commercial  data  sources.  In-service  safety  and  readiness  issues  are  grouped  by  priority  to  form  an 
integrated  picture  of  in-service  health,  operational  system  risk,  system  readiness,  and  future  in-service 
support  requirements. 

7.3  Tailoring  Reviews  to  Specific  Needs 

The  reviews  described  in  this  chapter  are  based  on  a  complex  system  development  project  requiring 
significant  technical  evaluation.  There  are  also  cases  in  which  system  technical  maturity  is  more 
advanced  than  normal  for  the  phase;  for  example,  a  situation  in  which  a  previous  program  or  an 
Advanced  Concept  Technology  Demonstration  (ACTD)  has  provided  a  significant  level  of  technical 
development  applicable  to  the  current  program.  In  some  cases,  tailoring  precipitates  the  merging  or 
even  elimination  of  acquisition  phases.  Tailoring  does  not  justify  the  elimination  of  technical 
management  activities  or  relieve  the  Government  PM  of  the  responsibilities  to  see  that  proper  SE 
disciplines  are  enforced.  It  does,  however,  highlight  the  need  for  flexibility  and  tailoring  to  the  specific 
needs  of  the  program  under  development. 

In  tailoring  efforts,  be  extremely  careful  determining  the  level  of  system  complexity.  The  system 
integration  effort,  the  development  of  a  single  advanced  technology  or  complex  subcomponent,  or  the 
need  for  intensive  software  development  may  be  sufficient  to  establish  the  total  system  as  a  complex 
project,  even  though  it  appears  simple  because  most  subsystems  are  simple  or  off-the-shelf. 
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7.4  Summary 

Technical  design  reviews  and  audits  are  an  integral  and  essential  part  of  the  SE  process.  The  meetings 
range  from  very  formal  reviews  by  Government  and  contractor  PMs  to  informal  technical  reviews 
concerned  with  product  or  task  elements  of  the  WBS.  Reviews  may  be  conducted  in  increments  over 
time.  All  reviews  share  the  common  objective  of  determining  the  technical  adequacy  of  the  existing 
design  to  meet  technical  requirements.  Reviews  and  audits  may  be  tailored  by  the  PM  to  support  the 
programs  requirements  and  objectives.  The  DT/OT  assessments  and  test  results  are  made  available  to 
the  reviews,  and  it  is  important  that  the  test  community  be  active  participants  in  a  programs  technical 
review  activities. 
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Test  &  Evaluation  Management  Guide 
Chapter  8  -  Integrated  Testing 

8.1  Introduction 

As  defined  in  the  DOT&E  and  Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Memorandum  (Reference  (aj)),  integrated  testing  is  the  collaborative  planning  and  collaborative 
execution  of  test  phases  and  events  to  provide  shared  data  in  support  of  independent  analysis, 
evaluation,  and  reporting  by  all  stakeholders,  particularly  the  developmental  (both  contractor  and 
Government)  and  operational  T&E  communities. 

This  chapter  discusses  the  use  of  integrated  testing  and  highlights  some  of  its  advantages  and 
disadvantages. 

8.2  Integrated  Testing 

The  goal  of  integrated  testing  is  to  conduct  a  seamless  test  program  that  produces  credible  qualitative 
and  quantitative  data  useful  to  all  evaluators  and  to  address  developmental,  sustainment,  and 
operational  issues  early  in  the  acquisition  process  to  the  decision  maker.  Even  with  the  best  integrated 
effort,  some  DT  must  inherently  be  sequential  as  it  is  inappropriate  or  unsafe  to  begin  OT  before  certain 
developmental  steps  and  capabilities  are  reached. 

Integrated  testing  allows  for  the  sharing  of  test  events,  in  which  a  single  test  point  or  mission  can 
provide  data  to  satisfy  multiple  objectives,  without  compromising  the  test  objectives  and  responsibilities 
of  participating  test  organizations.  Test  points  in  this  context  means  test  conditions  denoted  by  time, 
three-dimensional  location  and  energy  state,  and  system  operating  configuration,  in  which  preplanned 
test  techniques  are  applied  to  the  system  under  test  and  the  response(s)  are  observed  and  recorded. 
Integrated  testing  is  not  just  concurrent  or  combined  DT  and  OT,  in  which  both  DT  and  OT  test  points 
are  interleaved  on  the  same  mission  or  schedule.  Integrated  testing  focuses  the  entire  test  program 
(contractor  test.  Government  DT,  LET,  lA,  and  OT)  on  designing,  developing,  and  producing  a 
comprehensive  plan  that  coordinates  all  test  activities  to  support  evaluation  results  for  decision  makers 
at  required  decision  reviews. 

Although  these  ideas  of  integrated  testing  are  well  understood  in  theory,  their  actual  implementation 
has  been  far  more  difficult.  The  following  basic  principles  should  help  focus  the  efforts  of  the  T&E  and 
acquisition  communities: 

Collaboration  and  trust  are  required  on  two  levels:  between  the  acquisition  and  T&E  communities,  and 
among  all  test  organizations  (to  include  the  contractors)  who  work  on  the  program.  Without 
collaboration  and  trust,  T&E  will  remain  fragmented  and  suboptimized. 

Early  tester  involvement  and  influence  must  help  shape  the  emerging  requirements  and  acquisition 
strategies,  processes,  and  outputs.  The  earlier  integrated  testing  concepts  are  considered,  the  more 
they  will  benefit  the  program. 
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Integrated  testing  principles  and  methods  must  be  intentionally  designed  into  the  program,  preferably 
before  MS  A,  well  before  any  T&E  plans  are  written.  They  cannot  be  an  afterthought. 

Common  T&E  parameters,  methodologies,  and  terminology  must  be  agreed  upon  before  any  data  are 
collected  to  ensure  widest  possible  use  with  minimum  duplication. 

A  common  T&E  database  must  capture  the  data  in  ways  that  ensure  data  pedigree  and  integrity, 
validity,  security,  and  access  to  all  testers  as  much  as  possible. 

Integrated  test  planning  should  use  ST AT,  such  as  DOE  or  similar  tools,  to  make  each  test  event  serve 
multiple  goals  and  organizations.  Each  test  event  must  squeeze  out  as  much  value  and  data  as  possible. 

The  goals  of  each  test  organization  must  not  be  compromised;  each  organization  must  retain  its  ability 
to  report  results  to  decision  makers. 

The  amount  of  T&E  data  must  be  right  sized  to  the  needs  of  decision  makers  and  Warfighters  at  each 
program  waypoint.  Too  little  information  results  in  poor  execution,  and  too  much  produces  waste. 

It  is  just  as  important  to  define  what  integrated  testing  is  not.  Integrated  testing  is  not  an  event  or 
separate  test  phase,  and  it  is  not  a  new  type  of  test.  It  does  not  require  a  single  integrated  test  plan  but 
is  a  group  of  test  plans  that  are  integrated.  Integrated  testing  does  not  include  the  earliest  engineering 
testing  of  components  and  subsystems.  It  does  not  have  to  result  in  an  integrated  test  report  but  may 
have  several  reports  using  the  same  common  T&E  database. 

Integrated  testing  must  go  hand  in  glove  with  early  tester  involvement  or  influence-you  cannot  have  one 
without  the  other.  Early  influence  means  that  anyone  who  writes  or  reviews  early  documents  (i.e., 
requirements  documents,  acquisition  strategies,  theTES,  AoAs,  test  scenarios,  CONORS,  RFPs,  SOWs) 
must  use  integrated  testing  principles  and  goals  to  shape  these  forthcoming  program  documents.  The 
acquisition  strategy  and  first  formative  milestone  decision(s)  must  be  guided  by  these  goals  and 
principles;  otherwise,  the  insertion  of  integrated  testing  into  program  strategies  will  quickly  become 
more  difficult  over  time. 

Both  developmental  and  operational  testers  must  collaborate  during  the  requirements  development 
process  to  ensure  that  requirements  documents  are  formulated  so  all  testers  can  use  and  understand 
them.  These  requirements  must  be  translatable  into  contractual  language  for  contractor  testing;  CTPs 
for  DT;  and  COIs,  MOEs,  and  MOSs  for  operational  testers.  Acquisition  and  requirements  officials  must 
be  realistic  about  what  is  achievable  within  predetermined  technical,  time,  and  budget  constraints  and 
include  trade  space  for  delivery  of  capabilities  in  increments  if  necessary.  For  the  T&E  community, 
requirements  must  be  clear,  unambiguous,  testable,  and  operationally  relevant/realistic. 

8.3  Early  Involvement 

Early  information  about  achievement  of  performance  specifications  is  useful  to  the  decision  maker 
when  the  evaluation  and  information  are  provided  with  sufficient  time  to  react  and  affect  changes  in 
design.  The  key  point  is  that  saving  the  evaluation  of  DT  data  for  independent  evaluation  later  in  the 
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developmental  cycle  when  the  OT  occurs  negates  the  point  of  the  early  information.  As  stated  in  the 
MCOTEA  Manual  (Reference  (ak)),  information's  usefulness  diminishes  as  time  passes. 

As  soon  as  a  new  program  is  identified,  the  PM  should  form  a  T&E  WIPT  to  ensure  that  rigorous  T&E 
design  considerations  support  the  TES,  the  acquisition  strategy,  the  TEMP,  the  first  milestone  decision, 
and  any  other  early  T&E-related  activity.  The  T&E  WIPT  may  have  an  initial  cadre  of  T&E  members  who 
give  sound  advice  in  their  areas  of  expertise  using  any  available  early  information  about  the  proposed 
new  system  (e.g.,  technical  papers,  past  experience,  test  scenarios,  or  emerging  concepts  of 
employment).  The  PM  and  the  operational  tester  should  co-chair  the  T&E  WIPT,  which  ensures  T&E 
coverage  of  the  entire  program  from  the  earliest  to  the  latest  acquisition  phases.  Integrated  testing 
concepts  must  be  embedded  in  the  TES  and  TEMP. 

According  to  an  article  by  Edward  R.  Greer  in  the  ITEA  Journal,  integrated  testing  requires  careful 
planning  to  be  successful  and  has  multifaceted  meanings.  Some  of  these  meanings  include  the 
following: 

Integrated  testing  must  be  an  integral  part  of  development  and  acquisition,  providing  clear  and 
unambiguous  data.  All  stakeholder  needs  should  be  considered  and  met.  Stakeholders  include  the  PM, 
contractors,  the  program  management  team,  the  entire  development  team,  and  the  end  user. 

Contractor  and  Government  DT&E  must  be  planned  and  conducted  in  a  manner  such  that  there  is  no 
duplication  of  effort,  facilities,  personnel,  or  other  resources.  The  open  sharing  of  test  data  across  the 
test  spectrum,  from  contractor  to  Government  testing,  in  order  to  achieve  efficiencies  is  essential. 

The  continuance  of  a  smooth  and  integrated  flow  from  DT&E  with  and  into  OT&E  must  be  established. 
Figure  8-1  illustrates  an  integrated  testing  continuum  that  allows  for  efficiency  across  contractor  DT&E, 
Government  DT&E,  and  Government  OT&E. 

A  properly  developed  integrated  test  approach  can  generate  significant  time  and  cost  savings.  However, 
the  integrated  approach  must  not  compromise  either  DT  or  OT  objectives.  For  integrated  testing  to  be 
most  effective,  planning  efforts  must  be  carefully  coordinated  early  in  the  program  to  ensure  that  data 
are  obtained  to  satisfy  the  separate  needs  of  both  the  developing  agency  and  the  independent 
operational  tester.  Care  must  also  be  exercised  to  ensure  that  an  integrated  test  program  contains 
sufficient  OT  events  to  quantify  and  mitigate  risk  for  the  decision  maker  and  satisfy  the  requirement  for 
an  independent  evaluation.  In  all  integrated  test  programs,  provisions  for  separate  independent 
developmental  and  operational  evaluations  of  test  results  should  be  provided. 

The  T&E  continuum  illustrated  in  Figure  8-1  can  normally  be  divided  into  several  segments. 

In  the  initial  segment,  contractor  DT  events  usually  assume  priority  because  critical  technical  and 
engineering  tests  must  be  accomplished  to  continue  the  engineering  and  development  process.  During 
this  early  period,  OT  personnel  participate  to  gain  familiarity  with  the  system  and  to  gain  access  to  any 
test  data  that  can  support  OT  and  demonstrate  reliability  growth.  More  importantly,  OT  personnel  also 
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provide  operational  inputs  that  influence  system  design  early,  while  it  is  still  inexpensive  and  easy  to 
make  changes. 

Next,  the  integrated  portion  of  the  testing  frequently  includes  shared  DT  and  OT  objectives  or  joint  data 
requirements.  Again,  OT  involvement  continues  to  influencing  system  design. 

A  last  segment  may  contain  some  dedicated  and/or  separate  OT  events  to  be  conducted  primarily  by  the 
OTA.  The  OTA  and  implementing  command  must  ensure  that  the  integrated  test  is  planned  and 
executed  in  conjunction  with  the  top-level  evaluation  framework  matrix  in  the  TEMP  to  provide  the 
necessary  OT  information.  The  OTA  provides  an  independent  evaluation  of  the  system  and  is  ultimately 
responsible  for  achieving  OT&E  objectives. 
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Figure  8-1  Integrated  Testing  as  Part  of  an  Overall  T&E  Continuum 

Certain  test  events  can  be  organized  to  provide  information  useful  to  both  developmental  and 
operational  testers  through  implementation  of  a  coordinated  test  design  process.  One  or  more  STAT 
processes  could  be  used  to  develop  an  integrated  developmental  and  operational  test  program  that 
provides  confidence  that  the  performance  of  a  system  is  understood.  (DOE  is  one  such  methodology.) 
For  example,  prototype  free-fall  munitions  could  be  released  from  a  fighter  aircraft  under  operational 
employment  conditions  instead  of  from  a  static  stand  in  such  a  way  as  to  satisfy  DT  and  OT  objectives. 
Such  instances  need  to  be  identified  as  part  of  the  T&E  planning  process  to  prevent  duplication  of  effort. 
An  integrated  testing  approach  is  also  appropriate  for  certain  specialized  tests.  For  example,  in  the  case 
of  nuclear  hardness  and  survivability  (NFI&S)  testing,  systems  cannot  be  tested  in  a  totally  realistic 
operational  environment;  therefore,  a  single  NFI&S  test  may  be  used  to  meet  both  DT  and  OT  objectives. 
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8.4  Evaluation  Framework 

AT&E  program  strategy  should  be  structured  to  provide  knowledge  to  reduce  risk  in  acquisition  and 
operational  decisions.  That  knowledge  is  developed  through  the  evaluations  of  all  available  and  relevant 
data  and  information  from  contractor  and  Government  sources.  The  evaluation  framework  matrix 
describes  the  links  between  key  program  and  user  parameters  and  the  operational  and  developmental 
areas  that  must  be  evaluated  for  those  parameters.  It  correlates  the  knowledge  required  concerning 
KPPs/KSAs,  CTPs,  and  key  test  measures  (i.e.,  MOEs  and  MOSs)  and  the  planned  test  methods,  key  test 
resources,  and  facility  or  infrastructure  needs.  The  discussion  of  the  evaluation  framework  matrix  should 
also  identify  major  risks  or  limitations  to  completing  the  evaluations.  The  TEMP  reader  should  be  left 
with  a  clear  understanding  of  what  key  questions  the  evaluations  will  answer  for  the  program  and  user 
and  at  what  key  decision  points.  This  layout  and  discussion  will  also  provide  the  rationale  for  the  major 
test  objectives  and  the  resulting  major  resource  requirements  provided  in  the  Part  IV  of  the  TEMP. 

The  discussion  in  the  evaluation  framework  matrix  should  describe  the  intended  maturation  of  key 
technologies  and  the  overall  system,  the  evaluation  of  capabilities  in  a  mission  context,  and  evaluations 
needed  to  support  required  certifications  or  to  comply  with  statute.  The  details  of  how  the  evaluations 
will  be  performed  should  be  left  to  separate  evaluation  plans  (e.g.,  system  evaluation  plan  (Army),  OT&E 
plan,  LFT&E  plan). 

The  evaluation  of  the  maturation  of  a  system  or  capability  is  described  in  the  DT&E  section  of  the 
evaluation  framework  and  should  address  the  overall  approach  to  evaluate  development  of  system 
capabilities.  The  approach  should  cover  CTPs,  key  system  risks,  and  any  certifications  required  (weapon 
safety,  interoperability).  The  evaluation  of  technology  maturity  should  support  the  TDS.  The  evaluation 
of  system  maturity  should  support  the  acquisition  strategy.  The  extent  of  this  discussion  will  be  driven 
by  the  amount  of  development  in  the  acquisition  strategy.  For  example,  if  the  system  is  an  NDI  (i.e., 

COTS  or  Government  off-the-shelf  (GOTS)),  then  there  may  not  be  much,  if  any,  maturation  of  the 
system  required.  If  the  system  is  a  new  technologies  effort,  pushing  the  state  of  the  art  or  providing 
capabilities  significantly  improved  over  what  is  currently  being  achieved  in  the  operational  environment, 
then  there  may  be  a  significant  amount  of  effort  in  maturing  or  developing  the  system  or  its  support 
system  and,  therefore,  more  decisions  requiring  knowledge  from  evaluations.  In  assessing  the  level  of 
evaluations  necessary,  equal  consideration  should  be  given  to  the  maturity  of  the  technologies  used,  the 
degree  to  which  system  design  (hardware  and  software)  has  stabilized,  as  well  as  the  operational 
environment  for  the  employment  of  the  system.  A  lesson  learned  from  prior  programs  is  that  using  a 
COTS  item  in  a  new  environment  can  result  in  significant  capability  shortfalls  and,  therefore,  may 
require  unplanned  RDT&E  efforts  to  reach  required  maturity  levels. 

The  system  maturation  discussions  should  also  cover  evaluations  for  production  qualification, 
production  acceptance,  and  sustainment  of  the  system.  The  production  evaluations  may  be  covered  by 
DCMA  representatives  and  procedures  at  the  contractor's  manufacturing  plant,  or  they  may  require  T&E 
effort  to  establish  and  mature  the  processes.  Therefore,  the  appropriate  level  of  evaluation  could  range 
from  none,  for  normal  DCMA  practices,  to  minimal  for  first  article  qualification  checks,  to  more 
extensive  evaluations  based  upon  results  for  new  or  unique  manufacturing  techniques,  especially  with 
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new  technologies.  The  sustainment  evaluation  discussions  should  address  the  key  risks  or  issues  in 
sustaining  or  assessing  the  system  capability  in  operational  use.  The  sustainment  evaluation  discussion 
should  address  the  overall  logistics  T&E  effort,  maintenance  (both  corrective  and  preventative), 
servicing,  calibration,  and  support  aspects. 

The  discussion  of  mission  context  evaluations  addresses  the  approach  to  evaluate  operational 
effectiveness  and  operational  suitability  of  the  system  for  use  by  typical  users  in  the  intended  mission 
environments.  This  should  also  include  joint  operations  issues.  These  evaluations  provide  a  prediction  of 
how  well  the  system  will  perform  in  field  use  and  in  lOT&E  and  may  be  used  to  reduce  the  scope  of 
lOT&E,  but  these  evaluations  do  not  replace  or  eliminate  the  need  for  lOT&E. 

The  discussion  should  include  COIs.  COIs  are  the  operational  effectiveness  and  operational  suitability 
issues  (not  parameters,  objectives,  or  thresholds)  that  must  be  examined  to  evaluate/assess  the  systems 
capability  to  perform  its  mission.  Not  all  operational  issues  are  critical-COIs  must  be  relevant  to  the 
required  capabilities,  be  of  key  importance  to  the  system  being  operationally  effective  and  suitable,  and 
represent  a  significant  risk  if  not  satisfactorily  resolved. 

The  evaluation  strategy  must  include  those  evaluations  required  by  statute,  specifically  lOT&E, 
survivability,  and  lethality.  The  lOT&E  discussion  should  describe  the  approach  to  conduct  the 
independent  evaluation  of  the  system,  including  official  resolution  of  the  COIs.  The  discussion  of  the 
approach  to  evaluate  the  survivability/lethality  of  the  system  should  show  how  the  evaluation  will 
influence  the  development  and  maturation  of  the  system  design.  The  discussion  should  include  a 
description  of  the  overall  live-fire  evaluation  strategy  for  the  system  (as  defined  in  section  2366  of 
Reference  (b));  critical  live-fire  evaluation  issues;  and  any  major  evaluation  limitations. 

The  discussions  supporting  the  evaluation  framework  matrix  should  concisely  articulate  links  between 
key  decisions  in  the  system  life  cycle,  the  areas  that  need  to  be  evaluated  to  support  those  decisions, 
and  an  outline  of  the  test  methodologies  needed  to  obtain  the  data  for  the  evaluations.  The  discussions 
should  be  summarized  in  a  matrix  such  as  the  one  depicted  in  Figure  8-2,  which  shows  one  way  that  an 
evaluation  framework  matrix  can  be  organized.  Equivalent  Service-specific  formats  that  identify  the 
same  relationships  and  information  are  appropriate.  The  matrix  in  the  TEMP  is  an  executive-level  view 
that  should  provide  building  blocks  for  more  detailed  matrixes  and  links  within  supporting  test  plans. 
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Key  Requirements  and  T&E  Measures 

Test  Methodologies/  Key 
Resources  (M&S,  SIL,  MF, 

ISTF,  HITL,  OAR) 

Decision 

Supported 

Key  Reqs 

COIs 

Key  MOEs/MOSs 

CTPs  &  Threshold 

KPP  #1 : 

COI#1.  Is  the  XXX 
effective  for... 

MOE  1.1. 

Engine  Thrust 

Chamger  measurement 
Observation  of  performance 
profiles  OAR 

PDR 

CDR 

COI  #2.  Is  the  XXX 
suitable  for... 

Data  upload  time 

Component  level  replication 

Stress  and  Spike  testing  in  SIL 

PDR 

CDR 

COI  #3.  Can  the 

XXX  be... 

MOS  2.1. 

MS-C 

FRP 

MOE  1.3. 

Post-CDR 

FRP 

MOE  1.4 

Reliability  based  on 
growth  curve 

Component  level  stress  testing 
Sample  performance  on  growth 
curve 

Samipe  performance  with  M&S 
augmentation 

PDR 

CDR 

MS-C 

KPP#2 

MOS  2.4 

Data  link 

MS-C 

SR 

KSA#3 

COI  #4  Is  training... 

MOE  1.2. 

Observation  and  survey 

MS-C 

FRP 

KSA#3a 

COI  #5 

Documentation 

MOS  2.5 

MS-C 

FRP 

Figure  8-2  Top-Level  Evaluation  Framework  Matrix 

The  evaluation  framework  matrix  is  divided  into  three  sections:  Key  Requirements  and  T&E  Measures; 
Test  Methodologies/Key  Resources;  and  Decisions  Supported 

Key  Requirements  and  TStE  Measures  .  These  are  the  KPPs  and  KSAs  and  the  top-level  T&E  issues  and 
measures  for  evaluation.  The  top-level  T&E  issues  would  typically  include  COIs  and  COIC,  CTPs,  and  key 
MOEs/MOSs.  System-of-systems  COIs  should  also  be  included.  Each  measure  should  be  associated  with 
one  or  more  key  requirements.  However,  there  could  be  T&E  measures  without  an  associated  key 
requirement  or  COI/COIC.  Hence,  some  cells  in  the  evaluation  framework  matrix  may  be  empty.  A 
simple  test  to  determine  whether  this  section  of  the  matrix  is  minimally  adequate  is  to  confirm  that 
each  decision  supported  has  at  least  one  T&E  measure  associated  with  it,  and  that  each  key  requirement 
also  has  at  least  one  T&E  measure  associated  with  it.  Outside  of  that,  only  include  the  T&E  issues  and 
measures  that  drive  size  or  scope  of  the  T&E  program. 

Overview  of  Test  Methodologies  and  Key  Resources  .  These  identify  test  methodologies  or  key 
resources  necessary  to  generate  data  for  evaluations  to  support  decisions.  The  content  of  this  column 
should  indicate  the  key  methodologies  or  significant  resources  that  will  be  required.  Test  methodology 
refers  to  high-level  descriptions  of  methods  used  to  obtain  the  data.  For  example,  M&S,  system 
integration  lab,  or  open-air  range  each  represent  a  different  methodology  for  obtaining  test  data.  For 
situations  in  which  multiple  methodologies  are  acceptable,  show  the  preferred  methodology  that  will  be 
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used.  Short  notes  or  acronyms  should  be  used  to  identify  the  methodology.  Models  or  simulations 
should  be  identified  with  the  specific  name  or  acronym. 

Decisions  Supported  .  These  are  the  major  design,  developmental,  manufacturing,  programmatic, 
acquisition,  or  employment  decisions  driving  the  need  for  knowledge  to  be  obtained  through  T&E.  These 
decisions  include  acquisition  milestones/key  decision  points,  design  reviews,  certifications,  safety 
releases,  production  acceptance,  and  operational  employment/deployment.  The  operational 
employment/deployment  decisions  include  those  made  by  operators  and  maintainers  that  drive  the 
need  for  validated  operating  and  maintenance  manuals.  This  column  would  not  contain  each  decision 
that  an  operator  or  maintainer  would  make  but  just  the  overall  level  of  knowledge  needed  for  operating 
or  maintenance  data  or  instructions,  or  that  steer  significant  or  top-level  decisions.  The  key  determinant 
for  what  to  include  in  this  section  is  whether  the  decision  supported  (or  knowledge  requirement)  drives 
trade  space  for  performance,  cost  or  schedule,  or  the  size  or  scope  of  the  T&E  program.  Only  those 
decisions  that  facilitate  program  decisions  or  the  size  or  scope  of  the  T&E  program  should  be  included. 

8.5  Test  Data 

It  is  important  to  establish  common  T&E  tools,  methodologies,  standards,  formulas,  etc.,  that  enable  the 
widest  possible  utilization  of  all  data.  Commonality  cannot  be  an  afterthought  but  must  be  as  carefully 
planned  as  the  T&E  event  matrixes.  Data  annotated  in  furlongs  per  fortnight  would  be  difficult  to  merge 
together  with  data  annotated  in  miles  per  hour.  Agreement  on  a  common  T&E  database  that  has 
commonly  established  parameters  for  all  data  will  enhance  the  ease  and  usefulness  of  integrated 
testing. 

Careful  records  of  system  configuration  are  needed  to  help  all  stakeholders  decide  whether  they  can  use 
all  or  part  of  the  available  data.  Configurations  are  expected  to  change  during  DT&E,  and  not  all  DT&E 
data  may  be  usable  for  OT&E  purposes  because  OT&E  requires  data  from  production-representative  test 
articles.  However,  some  earlier  data  such  as  the  reliability  assessments  of  components  and  subsystems 
that  have  not  been  changed  may  still  be  usable  for  OT&E. 

8.6  Summary 

An  integrated  testing  approach  may  offer  an  effective  means  of  shortening  the  time  required  for  testing 
and  achieving  cost  savings.  Integrated  testing  is  the  DoDs  preferred  approach  to  structure  a  test 
program.  To  be  effective,  extensive  collaboration  is  required  to  ensure  that  the  developmental  and 
operational  test  requirements  are  addressed.  Key  to  a  successful  T&E  program  is  the  establishment  of 
the  T&E  WIPT.  It  is  possible  to  have  combined  test  teams,  consisting  of  DT,  OT,  and  contractor  personnel 
involved  throughout  the  testing  process.  The  teams  can  provide  mutual  support  and  share  mutually 
beneficial  data  as  long  as  the  test  program  is  carefully  planned  and  executed  and  reporting  activities  are 
conducted  separately. 
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Test  &  Evaluation  Management  Guide 
Chapter  9  -  Production  -  Related  Testing  Activities 

9.1  Introduction 

Most  of  the  T&E  activities  discussed  in  this  guide  concern  the  testing  of  the  actual  weapon  or  system 
being  developed.  However,  a  key  part  of  the  transition  to  production  involves  an  evaluation  of 
production-related  test  activities  and  the  production  process  that  manufactures  the  actual  system.  This 
chapter  describes  production  management  and  the  production  process  testing  that  can  be  performed  to 
assess  the  effectiveness  of  the  manufacturing  process  and  the  producibility  of  the  systems  design. 

Normally,  the  DT  and  OT  organizations  are  not  involved  directly  in  this  process.  Usually,  the 
manufacturing  and  quality  assurance  (QA)  sections  of  the  program  office  and  a  representative  of  the 
DCMA  oversee/perform  many  of  these  functions  and  can  assist  in  these  efforts.  DCMA  is  the  DoD 
Component  that  works  directly  with  defense  suppliers  to  help  ensure  that  DoD,  Federal,  and  allied 
Government  supplies  and  services  are  delivered  on  time  and  at  projected  cost  and  meet  all  performance 
requirements.  Before  contract  award,  DCMA  provides  advice  and  services  to  help  construct  effective 
solicitations,  identify  potential  risks,  select  the  most  capable  contractors,  and  write  contracts  that  meet 
customer  needs.  After  contract  award,  DCMA  monitors  contractor's  performance  and  management 
systems  to  ensure  that  cost,  product  performance,  and  delivery  schedules  are  in  compliance  with  the 
terms  and  conditions  of  the  contracts. 

9.2  Quality  In  Design 

Design  engineering  efforts  should  lead  to  a  producible  and  testable  product.  The  objectives  of  these 
design  efforts  are  to  achieve  effective  and  efficient  manufacturing  processes  with  the  necessary  process 
controls  to  satisfy  requirements  and  minimize  manufacturing  costs.  The  design  of  the  system  should 
facilitate,  throughout  the  supply  chain,  the  timely  and  affordable  manufacture,  assembly,  and  delivery 
of  a  quality  product  to  the  customer. 

QA  is  the  part  of  quality  management  focused  on  providing  confidence  that  quality  requirements  will  be 
fulfilled.  Effective  quality  management  activities  are  important  for  reducing  process-related  risks  to 
programs.  Further  information  about  quality  management  is  in  Reference  (j) . 

9.3  Production  Management 

Production  (or  manufacturing)  management  is  the  effective  use  of  resources  to  produce,  on  schedule, 
the  required  number  of  end  items  that  meet  specified  quality,  performance,  and  cost.  Production 
management  includes  but  is  not  limited  to  industrial  resource  analysis,  producibility  assessment, 
producibility  engineering  and  planning,  production  engineering,  industrial  preparedness  planning, 
postproduction  planning,  and  productivity  enhancement.  Production  management  begins  early  in  the 
acquisition  process,  as  early  as  the  concept  assessments,  and  is  specifically  addressed  at  each  program 
milestone  decision  point. 
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As  development  proceeds,  the  manufacturing  strategy  is  developed,  and  detailed  plans  are  made  for 
production.  Independent  producibility  assessments,  conducted  in  preparation  for  the  transition  from 
development  to  production,  are  reviewed  before  entering  LRIP.  Once  production  starts,  the  producibility 
of  the  system  design  concept  is  evaluated  to  verify  that  the  system  can  be  manufactured  in  compliance 
with  the  production-cost  and  the  industrial-base  goals  and  thresholds. 

The  LRIP  decision  is  supported  by  an  assessment  of  the  readiness  of  the  system  to  enter  production.  The 
system  cannot  enter  production  until  it  is  determined  that  the  principal  contractors  have  the  necessary 
resources  (i.e.,  physical,  financial,  and  managerial  capacity)  to  achieve  the  cost  and  schedule 
commitments  and  to  meet  peacetime  and  mobilization  requirements  for  production  of  the  system.  The 
method  of  formally  assessing  production  readiness  is  the  PRR  (See  Chapter  7),  which  is  conducted  prior 
to  MS  C. 

9.4  Role  Of  The  PRR 

The  PRR  is  intended  to  determine  the  status  of  completion  of  the  specific  actions  that  must  be 
satisfactorily  accomplished  before  executing  a  production  go-ahead  decision.  The  review  is  typically 
accomplished  in  an  incremental  fashion  before  commencing  production.  Usually  several  incremental 
reviews  and  one  final  review  are  conducted  to  assess  the  risk  in  exercising  the  production  go-ahead 
decision.  In  its  earlier  stages,  the  PRR  concerns  itself  with  gross-level  manufacturing  concerns  such  as 
the  need  for  identifying  high-risk/low-yield  manufacturing  processes  or  materials,  or  the  requirement 
for  manufacturing  development  effort  to  satisfy  design  requirements.  According  to  DAU  Technical  Guide 
to  Production  Readiness  Reviews  and  Manufacturing  Risk  Assessment ,  timing  of  the  incremental  PRRs  is 
event-driven  and  a  function  of  program  posture  and  is  not  specifically  locked  into  other  reviews. 

Figure  9-1  provides  a  list  of  typical  criteria  evaluated  as  part  of  a  PRR.  A  PRR  is  typically  conducted  by  an 
IPT,  which  may  include  DCMA  support.  The  IPT  lead  assembles  a  team  composed  of  individuals  in  the 
disciplines  of  design,  industry,  manufacturing,  procurement,  inventory  control,  contracts,  engineering, 
and  quality  training.  The  PRR  director  organizes  and  manages  the  team  effort  and  supervises 
preparation  of  the  findings.  PRR  decision  authority  lies  with  the  Government,  typically  the  PM.  Examples 
of  a  detailed  PRR  checklist  are  available  from  DAU  at 

https://acc.dau. mil/CommunitvBrowser.aspx?id=23248  .  The  checklist  may  be  scaled  down  based  on 
ACAT  level  or  program  risk. 

9.5  Production  Qualification  Testing 

Qualification  testing  is  performed  to  verify  the  design  and  manufacturing  process,  and  it  provides  a 
baseline  for  subsequent  acceptance  tests.  Production  qualification  testing  is  conducted  at  the  unit, 
subsystem,  and  system  level  on  production  items  and  is  completed  before  the  production  decision.  The 
results  of  these  tests  are  a  critical  factor  in  assessing  the  systems  readiness  for  production.  Down  line 
PQTs  are  performed  to  verify  process  control  and  may  be  performed  on  selected  parameters  rather  than 
at  the  levels  originally  selected  for  qualification. 


117 


525  &-yR9l3&lzuSy'a  l-yt-ESrSyuDcms 


/Kl-L04} 


Product  Design 

Producible  at  Low  Risk 

Stabilized  at  Low  Rate  of  Change 
Validated  Design 

Reliability,  Maintainability,  and  Performance  Demonstrated 
Industrial  Resources 

Adequate  Plant  Capacity  (Peacetime  and  Wartime  Demands) 

Facilities,  Special  Production  and  Test  Equipment,  and  Tooling  Identified 
Needed  Plant  Modernization  (CAD/CAf\1,  Other  Automation)  Accomplished 
Associated  Computer  Software  Developed 
Skilled  Personnel  and  Training  Programs  Available 

Production  Engineering  and  Planning 
Production  Plan  Developed 

Production  Schedules  Compatible  with  Delivery  Requirements 

Manufacturing  Methods  and  Processes  Integrated  with  Facilities,  Equipment,  Tooling  and 
Plant  Layout 

Value  Engineering  Applied 

Alternate  Production  Approaches  Available 

Drawings,  Standards  and  Shop  Instructions  are  explicit 

Configuration  Management  Adequate 

Production  Policies  and  Procedures  Documented 

Management  Information  System  Adequate 

Contractors  Management  Structure  is  Acceptable  to  the  PMO 

Materials 

All  Selected  Materials  Approved  By  Contractors  Material  Engineers 

Bill  of  Materials  Prepared 

Make-or-Buy  Decisions  Complete 

Procurement  of  Long  Lead-Time  Items  Identified 

Sole-Source  and  Government-Furnished  Items  Identified 

Contractors  Inventory-Control  System  Adequate 

Contractors  Material  Cost  Procurement  Plan  Complete 

Quality  Assurance  (QA) 

Quality  Plan  in  Accordance  with  Contract  Requirements 
Quality  Control  Procedures  and  Acceptance  Criteria  Established 
QA  Organization  Participates  in  Production  Planning  Effort 

Logistics 

Operational  Support,  Test  and  Diagnostic  Equipment  Available  at  System  Deployment 
Training  Aids,  Simulators,  and  Other  Devices  Ready  at  System  Deployment 
Spares  Integrated  into  Production  Lot  Flow 


Figure  9-1  Typical  Criteria  Evaluated  as  Part  of  a  PRR 
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9.5.1  PQTs 

PQTs  are  a  series  of  formal  contractual  tests  conducted  to  ensure  design  integrity  over  the  specified 
operational  and  environmental  range.  The  tests  are  conducted  on  pre-FRP  items  fabricated  to  the 
proposed  production  design  drawings  and  specifications.  The  PQTs  include  all  contractual  R&M 
demonstration  tests  required  before  production  release.  For  volume  acquisitions,  these  tests  are  a 
constraint  to  production  release. 

9.5.2  First  Article  Tests  (FATs) 

FATs  consist  of  a  series  of  formal  contractual  tests  conducted  to  ensure  the  effectiveness  of  the 
manufacturing  process,  equipment,  and  procedures.  These  tests  are  conducted  on  a  random  sample 
from  the  first  production  lot.  According  to  Subpart  9.3  of  the  Federal  Acquisition  Regulation  (Reference 
(al)),  these  series  of  tests  are  repeated  if  the  manufacturing  process,  equipment,  or  procedure  is 
changed  significantly  and  when  a  second  or  alternative  source  of  manufacturing  is  brought  online. 

9.6  Transition  To  Production 

In  an  acquisition  process,  often  the  first  indication  that  a  system  will  experience  problems  is  during  the 
transition  from  engineering  design  to  LRIP.  This  transition  may  continue  over  an  extended  period,  and 
during  this  period,  the  system  is  undergoing  stringent  contractor  and  Government  testing.  There  may  be 
unexpected  failures  requiring  significant  design  changes,  which  impact  quality,  producibility,  and 
supportability  and  may  require  program  schedule  slippage.  Long  periods  of  transition  usually  indicate 
that  insufficient  attention  to  design  or  producibility  was  given  early  in  the  programs  acquisition  process. 

9.6.1  Transition  Planning 

Producibility  engineering  and  planning  is  the  common  thread  that  guides  a  system  from  early  concept  to 
production.  Planning  is  a  management  tool  used  to  ensure  that  adequate  risk-handling  measures  have 
been  taken  to  transition  from  development  to  production.  One  product  from  the  planning  process  is  a 
checklist  for  use  during  the  readiness  reviews.  Planning  should  tie  together  the  applications  of 
designing,  testing,  and  manufacturing  activities  to  reduce  data  requirements,  duplication  of  effort,  costs, 
and  scheduling  and  to  ensure  early  success  of  the  LRIP  first  production  article. 

9.6.2  Testing  During  the  Transition 

Testing  accomplished  during  the  transition  from  development  to  production  will  include  acceptance 
testing,  manufacturing  screening,  and  final  testing.  These  technical  tests  are  performed  by  the 
contractor  to  ensure  that  the  system  will  transition  smoothly  and  that  test  design  and  manufacturing 
issues  affecting  design  are  addressed.  During  this  same  period,  the  Government  will  be  using  the  latest 
available  CIs  to  conduct  the  lOT&E.  The  impact  of  these  tests  may  overwhelm  the  configuration 
management  of  the  system  unless  careful  planning  is  accomplished  to  handle  these  changes. 
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9.7  LRIP 

LRIP  is  the  production  of  a  system  in  limited  quantity  to  provide  articles  for  lOT&E  and  to  demonstrate 
production  capability.  Also,  LRIP  permits  an  orderly  increase  in  the  production  rate  sufficient  to  lead  to 
FRP  upon  successful  completion  of  OT.  The  decision  to  have  an  LRIP  is  made  at  the  MS  C  approval  of  the 
program  acquisition  strategy.  At  that  time,  the  PM  must  identify  the  quantity  to  be  produced  during 
LRIP  and  validate  the  quantity  of  LRIP  articles  to  be  used  for  lOT&E  (ACAT  I  approved  by  the  DOT&E; 
ACAT  II  and  III  approved  by  the  Service  OTA)  in  accordance  with  Enclosure  2  of  Reference  (d). 

Consistent  with  section  2400  of  Reference  (b).  Reference  (d),  and  Reference  (h),  production- 
representative  articles  should  be  used  for  T&E.  Wherever  practical,  T&E  will  be  conducted  using  LRIP 
systems  assembled  using  the  parts,  tools,  and  manufacturing  processes  intended  for  use  in  FRP.  The 
system  will  also  utilize  the  intended  production  versions  of  software.  In  addition,  the  logistics  system 
and  maintenance  manuals  intended  for  use  on  the  fielded  system  should  be  in  place. 

When  the  use  of  LRIP  articles  is  impractical,  the  system  used  in  T&E  should,  at  a  minimum,  incorporate 
the  same  parts  and  software  to  be  used  in  LRIP  articles.  In  particular,  the  hardware  and  software  should 
be  as  defined  by  the  system-level  CDR,  FCA,  and  SVR,  including  correction  of  appropriate  major 
deficiencies  identified  during  DT.  (See  Chapter  7  of  this  Guide  for  a  discussion  of  the  technical  reviews.) 
Manufacturing  processes  to  be  used  in  FRP  should  also  be  adhered  to  as  closely  as  possible.  DOT&E 
must  be  provided  with  detailed  information  describing  any  process  differences  in  order  to 
independently  evaluate  whether  the  differences  are  acceptable. 

DOT&E  will  assess  adherence  to  the  above  guidelines  as  part  of  its  responsibility  for  reviewing  and 
approving  TEMPs  and  test  plans.  Proposals  to  use  articles  not  from  LRIP  to  conduct  lOT&E  will  be 
considered  for  approval  by  DOT&E  using  the  criteria  stated  above  as  part  of  those  reviews. 

When  the  decision  authority  thinks  the  systems  will  not  perform  to  expectation,  the  PM  may  direct  that 
the  system  not  proceed  into  LRIP  until  there  is  a  program  review.  The  DOT&E  submits  a  BLRIP  report  on 
all  oversight  systems  to  congressional  committees  before  the  FRP  decision,  which  approves  the  system 
to  proceed  beyond  LRIP,  is  made. 

9.8  PAT&E 

The  PAT&E  ensures  that  production  items  demonstrate  the  fulfillment  of  the  requirements  and 
specifications  of  the  procuring  contract  or  agreements.  The  testing  also  ensures  that  the  system  being 
produced  demonstrates  the  same  performance  as  the  pre-FRP  models.  The  procured  items  or  system 
must  operate  in  accordance  with  system  and  item  specifications.  The  PAT&E  is  usually  conducted  by  the 
program  office  QA  section  at  the  contractor's  plant  or  by  DCMA  support  staff  and  may  involve 
operational  users.  Acceptance  may  be  either  at  the  source  (contractor's  plant)  or  the  destination 
(Government-identified  site)  based  on  the  need  for  test  or  inspection  resources  that  may  not  be  readily 
available  at  the  contractor's  facilities.  At  a  high  level,  PAT&E  prescribes  the  policies  and  procedures  to 
ensure  that  supplies  and  services  conform  to  contract  requirements  in  Subpart  46.401  of  Reference  (al). 


120 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcms 


/Kl-L04} 


The  effectiveness  of  a  PAT&E  program  is  directly  tied  to  inclusion  of  contract  language  that  clearly 
defines  the  acceptance  requirements  and  outcomes  of  compliance.  Appropriate  contractual  acceptance 
criteria  are  developed  by  understanding  program  reliability  and  performance  requirements  and 
identifying  the  methods  and  thresholds  by  which  the  compliance  to  requirements  will  be  evaluated.  In 
accordance  with  the  acquisition  regulation,  incentives  must  be  used  to  the  maximum  extent  practical 
and  may  be  positive,  negative,  or  a  combination  of  both.  Government  surveillance  (PAT&E)  is  usually 
formalized  in  a  quality  assurance  surveillance  plan  (QASP)  (See  Subpart  46.401  of  Reference  (al)).  The 
QASP  is  incorporated  into  the  contract. 

For  the  PAT&E  of  an  Air  Force  bomber,  for  example,  the  contractor  and  Government  QA  inspectors 
reviewed  all  manufacturing  and  ground  testing  results  for  each  aircraft.  In  addition,  a  flight  test  team, 
composed  of  contractor  and  Air  Force  test  pilots,  flew  each  aircraft  a  minimum  of  10  hours, 
demonstrating  all  onboard  aircraft  systems  while  in  flight.  Any  discrepancies  in  flight  were  noted, 
corrected,  and  tested  on  the  ground;  they  were  then  retested  on  subsequent  checkouts  and  acceptance 
flights.  Once  each  aircraft  had  passed  all  tests  and  all  systems  were  fully  operational.  Air  Force 
authorities  accepted  the  aircraft.  The  test  documentation  also  became  part  of  the  delivered  package. 
During  this  test  period,  the  program  office  monitored  each  aircrafts  daily  progress. 

9.9  Summary 

A  primary  purpose  of  production-related  testing  is  to  lower  the  production  risk.  The  PM  must  ensure 
that  the  contractors  and  subcontractor's  manufacturing  strategy  and  capabilities  will  result  in  the 
desired  product  within  acceptable  cost.  The  LRIP  and  PAT&E  also  play  major  roles  in  ensuring  that  the 
production  unit  is  identical  to  the  design  drawings  and  conforms  to  the  specifications  of  the  contract, 
the  lOT&E  is  conducted  with  representative  system  configurations,  and  the  system  fielded  will  meet  the 
user's  needs. 
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Test  &  Evaluation  Management  Guide 

Chapter  10  -  Introduction  to  Operational  Test  and  Evaluation 

10.1  Introduction 

This  chapter  provides  an  introduction  to  the  concept  of  OT&E.  it  outiines  the  purpose  of  OT&E,  defines 
OT&E,  discusses  the  primary  participants  in  the  OT&E  process,  describes  severai  types  of  OT&E,  and 
inciudes  some  generai  guideiines  for  the  successfui  pianning,  execution,  and  reporting  of  OT&E 
programs. 

10.2  Purpose  and  Scope  of  OT&E 

OT&E  is  conducted  for  major  programs  by  an  organization  that  is  independent  of  the  deveioping, 
procuring,  and  using  commands.  Some  form  of  OA  or  OT  is  normaiiy  conducted  in  each  acquisition 
phase.  Each  shouid  be  keyed  to  a  decision  review  in  the  materiei  acquisition  process.  OT&E  shouid  be 
focused  on  mission  accompiishment  rather  than  meeting  specifications,  requirements,  MOEs,  and 
MOPs.  The  OT&E  provides  the  decision  authority  with  an  estimate  of  the  following: 

•  The  degree  of  satisfaction  of  the  user's  requirements  expressed  as  operational  effectiveness  and 
operational  suitability  of  the  new  system. 

•  The  systems  current  capabilities,  considering  equipment  already  available  and  operational 
benefits  or  burdens  associated  with  the  new  system. 

•  The  need  for  further  development  of  the  new  system  to  correct  performance  deficiencies. 

•  The  adequacy  of  doctrine,  organizations,  operating  techniques,  tactics,  and  training  for 
employment  of  the  system;  of  maintenance  support  for  the  system;  and  of  the  systems 
performance  in  the  countermeasures  environment. 

OT&E  is  legally  defined  in  section  2399  of  Reference  (b)  as  the  field  test,  under  realistic  combat 
conditions,  of  any  item  of  (or  key  component  of)  weapons,  equipment,  or  munitions  for  the  purposes  of 
determining  the  effectiveness  and  suitability  of  the  weapons,  equipment,  or  munitions  for  use  in 
combat  by  typical  military  users;  and  the  evaluation  of  the  results  of  such  test.  This  term  does  not 
include  an  operational  assessment  based  exclusively  on  computer  modeling;  simulation;  or  an  analysis 
of  system  requirements,  engineering  proposals,  design  specifications,  or  any  other  information 
contained  in  program  documents. 

Key  aspects  of  this  legal  definition  of  OT&E  are  the  terms  operational  effectiveness  and  operational 
suitability,  which  can  be  defined  as  follows: 

•  Operational  Effectiveness  .  The  degree  to  which  the  system  accomplishes  its  missions  when 
employed  by  operational  personnel  in  a  realistic  scenario  (natural,  electronic,  threat)  with  the 
appropriate  organization,  doctrine,  supportability,  survivability,  vulnerability,  and  threat 
(including  countermeasures,  nuclear  threats,  nuclear  effects,  and  NBC  threats)  environment, 
and  using  tactics  and  techniques  developed  during  earlier  FOT&E. 
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•  Operational  Suitability  .  The  degree  to  which  the  system  can  be  placed  in  operational  field  use, 
with  specific  evaluations  of  availability,  compatibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors,  manpower 
supportability,  natural  environmental  effects  and  impacts,  logistics  supportability,  and 
documentation  and  training  requirements. 

Operational  effectiveness  and  suitability  must  be  evaluated  and  reported  on  the  basis  of  whether  a 
system  can  be  used  by  Soldiers,  Sailors,  Airmen,  and  Marines  to  accomplish  a  combat  mission.  The 
appropriate  environment  for  that  evaluation  includes  the  system  under  test  and  all  interrelated  systems 
(i.e.,  its  planned  or  expected  environment  in  terms  of  weapons,  sensors,  C2,  and  platforms,  as 
appropriate)  needed  to  accomplish  an  end-to-end  mission  in  combat.  The  data  used  for  evaluation  are 
appropriately  called  MOEs  because  they  measure  the  military  effect  (mission  accomplishment)  that 
comes  from  the  use  of  the  system  in  its  expected  environment.  Figure  10-1  depicts  the  scope  of  OT&E. 


Figure  10-1  Scope  of  the  OT&E  Effort 
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10.3  Test  Participants 

OT&E  is  managed  by  an  independent  OTA,  which  each  Service  is  required  to  maintain.  Personnel  who 
operate,  maintain,  and  support  the  system  during  OT&E  are  trained  to  a  level  commensurate  with  that 
of  typical  personnel  who  will  perform  these  functions  under  peacetime  and  wartime  conditions.  Also, 
PMO  personnel,  the  IPTs,  and  test  coordinating  groups  play  important  parts  in  the  overall  OT&E 
planning  and  execution  process. 

10.3.1  Service  OTAs 

The  OT&E  agencies  should  become  involved  early  in  the  systems  life  cycle,  usually  during  the  programs 
evaluation  of  concepts.  At  that  time,  they  can  begin  to  develop  strategies  for  conducting  OT&E.  As  test 
planning  continues,  a  more  detailed  TEMP  (which  includes  OTs,  among  other  tests)  is  developed,  and 
the  test  resources  are  identified  and  scheduled.  During  the  early  stages,  the  OTAs  structure  an  OT&E 
program  consistent  with  the  approved  acquisition  strategy  for  the  system,  identify  critical  OT  issues,  and 
assess  the  adequacy  of  candidate  systems.  As  the  program  moves  into  advanced  planning,  OT&E  efforts 
are  directed  toward  becoming  familiar  with  the  system,  encouraging  interface  between  the  user  and 
developer,  and  further  refining  the  COIs.  The  OTA  test  directors,  analysts,  and  evaluators  design  the 
OT&E  so  that  the  data  collected  will  support  answering  the  COIs. 

Each  Service  has  an  independent  organization  dedicated  to  planning,  executing,  and  reporting  the 
results  of  that  Service's  OT&E  activities.  These  organizations  are  ATEC,  Navy  OPTEVFOR,  AFOTEC,  and 
MCOTEA.  Although  policy  requires  an  OTA  only  within  the  Services,  DISA  has  designated  JITC  as  the  OTA 
for  its  DoD  information  systems  and  a  special  operations  command  element  conducts  many  of  its  own 
lOT&Es. 

10.3.2  Test  Personnel 

OT  is  conducted  on  materiel  systems  with  typical  user  organizational  units  in  a  realistic  operational 
environment.  It  uses  personnel  with  the  same  specialties  (i.e.,  military  occupational  specialties.  Air  Force 
specialty  codes)  as  those  who  will  operate,  maintain,  and  support  the  system  when  fielded.  Participants 
are  trained  in  the  systems  operation  based  on  the  Services  operational  mission  profiles.  Because  some 
OT&E  consists  of  force-on-force  tests,  the  forces  opposing  the  tested  system  must  also  be  trained  in  the 
use  of  threat  equipment,  tactics,  and  doctrine.  For  OT  conducted  before  lOT&E,  most  system  training  is 
conducted  by  the  systems  contractor.  For  lOT&E,  the  contractor  trains  the  Service  school  cadre,  who 
then  train  the  participating  organizational  units.  Once  the  system  has  entered  FRP,  the  Service  will 
normally  assume  training  responsibilities,  unless  the  contractor  will  continue  to  train  once  the  system  is 
fielded.  OT  often  requires  a  large  support  staff  of  data  collectors  and  scenario  controllers  operating  in 
the  field  with  the  user  test  forces  and  opposing  forces. 
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10.4  OT&E  as  it  Relates  to  DT&E 

As  shown  in  Figure  10-2,  DT  is  focused  on  verifying  that  the  system  under  test  meets  its  technical 
requirements,  and  OT  focuses  on  validating  the  functioning  of  the  equipment  in  a  realistic  combat 
environment.  Although  DT&E  and  OT&E  are  separate  activities  and  are  conducted  by  different  test 
communities,  the  communities  frequently  interact  and  complement  each  other.  Well-constructed  test 
programs  can  and  should  be  integrated  in  many  respects.  DT&E  provides  a  view  (verification)  of  the 
technical  objectives,  and  OT&E  provides  an  assessment  (validation)  of  the  systems  potential  to  satisfy 
user  requirements. 


DT 

lOT&E 

•  Controlled  by  program  office 

•  Controlled  by  independent  agency 

•  One-on-one  tests 

•  Many-on-many  tests 

•  Controlled  environment 

•  Realistic  environment  vifith  operational  scenario 

•  Contractor  involvement 

•  Restricted  contractor  involvement 

•  Trained,  experienced  operators 

•  User  troops  recently  trained 

•  Precise  Performance  Objectives  and  Threshold 

•  Performance  Measurement  of  Operational  Effectiveness  & 

Measurement 

Suitability 

•  T est  to  Technical  Specification 

•  Test  to  operational  requirements 

•  Development  T est  Article 

•  Production  representative  test  article 

Figure  10-2  Comparison  of  DT  and  lOT&E  Differences  Highlighted 
10.5  Types  of  OT&E 

OT&E  efforts  can  be  divided  into  two  phases:  (1)  OT  performed  before  FRP  and  (2)  OT  performed  after 
the  FRP  decision.  The  pre-FRP  OT&E  includes  EOAs,  OAs,  and  lOT&E.  OAs  begin  early  in  the  program, 
frequently  before  program  start,  and  continue  until  the  system  is  certified  as  ready  for  lOT&E.  The 
lOT&E  is  conducted  late  during  low-rate  production,  when  test  assets  are  available,  in  support  of  the 
FRPDR.  The  Navy  uses  the  term  OPEVAL  (operational  evaluation)  to  define  lOT&E.  After  transition  to 
FRP,  all  subsequent  OT  is  usually  referred  to  as  FOT&E. 

Evolutionary  acquisition  is  the  DoDs  preferred  acquisition  approach  for  many  defense  systems.  An 
evolutionary  approach  delivers  capability  in  increments,  recognizing  upfront  the  need  for  future 
capability  improvements.  Each  increment  is  a  militarily  useful  and  supportable  operational  capability 
that  can  be  developed,  produced,  deployed,  and  sustained.  Block  upgrades,  P3ls,  and  similar  efforts  that 
provide  a  significant  increase  in  operational  capability  and  meet  an  acquisition  category  threshold  as 
specified  by  Reference  (d)  are  managed  as  separate  increments.  Early  involvement  of  all  test  agencies  is 
essential  in  the  first  and  subsequent  evolutionary  increment.  Test  content  and  reporting  should  be 
tailored  against  earlier  test  results,  evaluating  at  a  minimum  the  increment  of  mission  accomplishment 
and  survivability  required  of  the  new  increment,  plus  whether  performance  previously  demonstrated  by 
the  previous  increment  has  been  degraded  in  accordance  with  Enclosure  6  of  Reference  (d). 
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10.5.1  EGAs 

EGAs  are  conducted  primarily  to  forecast  and  evaluate  the  potential  operational  effectiveness  and 
suitability  of  the  weapon  system  during  development.  EGAs  started  during  the  MSA  and/or  TD 
acquisition  phases  may  continue  into  system  integration  and  are  conducted  on  prototypes  of  the 
developing  design. 

10.5.2  OAs 

An  GA  is  an  evaluation  of  operational  effectiveness  and  operational  suitability  made  by  an  independent 
GT  activity,  with  user  support  as  required,  on  other  than  production  systems.  The  focus  of  an  GA  is  on 
significant  trends  noted  in  development  efforts,  programmatic  voids,  risk  areas,  adequacy  of 
requirements,  and  the  ability  of  the  program  to  support  adequate  GT.  An  GA  may  be  conducted  at  any 
time  using  technology  demonstrators,  prototypes,  mock-ups,  EDMs,  or  simulators  but  will  not  substitute 
for  the  IGT&E  necessary  to  support  FRP  decisions.  An  GA  is  normally  conducted  prior  to  MS  C. 

The  GA  normally  takes  place  during  each  phase  of  the  acquisition  process  prior  to  IGT&E.  It  is  used  to 
provide  an  early  assessment  of  potential  operational  effectiveness  and  suitability  for  decision  makers  at 
decision  points.  These  assessments  attempt  to  project  the  systems  potential  to  meet  the  user's 
requirements.  Assessments  conducted  early  in  the  program  development  process  may  be  called  EGAs. 

The  GA  begins  when  the  GTAs  start  their  evaluations  of  system-level  performance.  The  GTA  uses  any 
testing  results,  M&S,  and  data  from  other  sources  during  an  assessment.  These  data  are  evaluated  by 
the  GTA  from  an  operational  point  of  view.  As  the  program  matures,  these  GAs  of  performance 
requirements  are  conducted  on  EDMs  or  pre-production  articles  until  the  system  performance  is 
considered  mature.  The  mature  system  can  then  be  certified  ready  for  its  IGT&E  (GPEVAL  in  the  Navy). 
When  using  an  evolutionary  acquisition  strategy,  subsequent  increments  will  at  least  have  an  GA  before 
the  new  increment  is  issued  to  the  user. 

10.5.3  lOT&E  (Navy  OPEVAL) 

Prior  to  entering  IGT&E,  an  GTRR  is  conducted.  An  GTRR  is  a  multi-disciplined  product  and  process 
assessment  to  ensure  that  the  system  can  proceed  into  IGT&E  with  a  high  probability  of  success  and 
that  the  system  is  effective  and  suitable  for  service  introduction.  The  FRP  decision  may  hinge  on  this 
successful  determination.  The  understanding  of  system  performance  in  the  operational  environment  to 
meet  the  CPD  is  important  to  the  GTRR.  Consequently,  it  is  important  that  the  test  addresses  and 
verifies  system  reliability,  maintainability,  and  supportability  performance  and  determines  whether  the 
hazards  and  ESGFI  residual  risks  are  manageable  within  the  planned  testing  operations.  The  GTRR  is 
complete  when  the  SAE,  or  the  appointed  MDA,  evaluates  and  determines  materiel  system  readiness  for 
IGT&E. 

The  GT&E  performed  in  support  of  the  FRP  decision  is  generally  known  as  IGT&E.  The  Navy  calls  this 
event  GPEVAL.  The  IGT&E  occurs  during  LRIP  and  must  be  completed  before  the  FRPDR.  More  than  one 
IGT&E  may  be  conducted  on  the  system  if  there  are  system  performance  problems  requiring  retest,  the 
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system  is  decertified,  or  a  need  exists  to  test  in  additional  environments.  The  lOT&E  is  conducted  on  a 
production  or  production-representative  system  using  typical  operational  personnel  in  a  realistic 
combat  scenario. 

The  lOT&E  is  the  final  dedicated  phase  of  OT&E  preceding  an  FRP  decision.  The  Services  OTRR  precedes 
the  beginning  of  testing  considers  the  results  of  DT&E,  manufacturing  processes,  safety,  and  other 
Service  specific  exit/success  criteria.  For  OSD  T&E  oversight  programs,  the  DASD(DT&E)  submits  an 
AOTR  for  use  in  the  certification  of  readiness  for  OT&E  decision.  The  lOT&E  is  conducted  by  an  OT&E 
agency  independent  of  the  contractor,  PMO,  or  developing  agency.  lOT&E  has  been  described  as 
follows: 


•  All  OT&E  conducted  on  production  or  production-representative  articles  to  support  the  decision 
to  proceed  beyond  LRIP.  It  is  conducted  to  provide  a  valid  estimate  of  expected  system 
operational  effectiveness  and  operational  suitability.  The  definition  of  OT&E  as  spelled  out  in 
section  139  of  Reference  (b)  is  generally  considered  applicable  only  to  lOT&E. 

•  The  results  from  lOT&E  are  evaluated  and  presented  to  the  MDA  to  support  the  FRP  decision, 
and  for  MDAPs  and  selected  OSD  oversight  programs,  the  results  are  documented  in  DOT&Es 
BLRIP  report.  This  phase  of  OT&E  addresses  the  KPPs  identified  in  the  CPD  and  the  COIs 
identified  in  the  TEMP.  lOT&E  plans  for  ACAT  I  and  lA  and  other  designated  programs  must  be 
approved  by  the  DOT&E.  DOT&E  considers  the  Service  reports,  but  the  foundation  of  the  BLRIP 
report  is  DOT&Es  own  independent  analysis  and  evaluation. 

10.5.4  FOT&E 

FOT&E  is  the  OT&E  that  may  be  necessary  after  the  FRPDR  to  refine  the  estimates  made  during  lOT&E, 
to  evaluate  changes,  and  to  reevaluate  the  system  to  ensure  that  it  continues  to  meet  operational  needs 
and  retains  its  effectiveness  in  a  new  environment  or  against  a  new  threat.  FOT&E  is  conducted  during 
fielding/deployment  and  operational  support  and  sometimes  may  be  divided  into  two  separate 
activities.  Preliminary  FOT&E  is  normally  conducted  after  IOC  is  attained  to  assess  full  system  capability. 
It  is  conducted  by  the  OTA  or  designated  organization  to  verify  the  correction  of  deficiencies,  if  required, 
and  to  assess  system  training  and  logistics  status  not  evaluated  during  lOT&E.  Subsequent  FOT&E  is 
conducted  on  production  items  throughout  the  life  of  a  system.  The  results  are  used  to  refine  estimates 
of  operational  effectiveness  and  suitability;  to  update  training,  tactics,  techniques,  and  doctrine;  and  to 
identify  operational  deficiencies  and  evaluate  modifications.  This  later  FOT&E  often  is  conducted  by  the 
operating  command. 

FOT&E  is  conducted  in  a  realistic  tactical  environment  similar  to  that  used  in  lOT&E,  but  fewer  test  items 
may  be  used.  Normally,  FOT&E  is  conducted  using  fielded  production  systems.  Specific  objectives  of 
FOT&E  include  testing  modifications  that  are  to  be  incorporated  into  production  systems,  completing 
any  deferred  or  incomplete  lOT&E,  evaluating  correction  of  deficiencies  found  during  lOT&E,  and 
assessing  reliability  including  spares  support  on  deployed  systems.  The  tests  are  also  used  to  evaluate 
the  system  in  a  different  platform  application  for  new  tactical  applications  or  against  new  threats  . 


127 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcmS 


/Kl-L0JM 


10.5.5  Qualification  Operational  Test  and  Evaluation  (QOT&E) 

QOT&E  may  be  performed  by  the  major  command,  user,  or  OTA  and  is  conducted  on  minor 
modifications  or  new  applications  of  existing  equipment  when  no  R&D  funding  is  required.  An  example 
of  a  program  in  which  QOT&E  was  performed  by  the  Air  Force  is  the  A-10  Air-to-Air  Self-Defense 
Program.  In  this  program,  the  mission  of  the  A-10  was  expanded  from  strictly  ground  support  to  include 
an  air-to-air  defense  role.  To  accomplish  this  expansion,  the  A-10  aircraft  was  modified  with  off-the- 
shelf  AIM-9  and  air-to-air  missiles;  QOT&E  was  performed  on  the  system  to  evaluate  its  operational 
effectiveness  and  suitability. 

10.6  Test  Planning 

OT  planning  is  one  of  the  most  important  parts  of  the  OT&E  process.  Proper  planning  facilitates  the 
acquisition  of  data  to  support  the  determination  of  the  weapon  systems  operational  effectiveness  and 
suitability.  Planning  must  be  pursued  in  a  deliberate,  comprehensive,  and  structured  manner.  Careful 
and  complete  planning  may  not  guarantee  a  successful  test  program,  but  inadequate  planning  can  result 
in  significant  test  problems,  poor  system  performance,  and  cost  overruns.  OT  planning  is  conducted  by 
the  OTA  before  program  start,  and  more  detailed  planning  usually  starts  about  2  years  before  each  OT 
event. 

Operational  planning  can  be  divided  into  three  phases:  early  planning,  advanced  planning,  and  detailed 
planning.  Early  planning  entails  developing  COIs,  formulating  a  plan  for  evaluations,  determining  the 
CONOPS,  envisioning  the  operational  environment,  and  developing  mission  scenarios  and  resource 
requirements.  Advanced  planning  determines  the  purpose  and  scope  of  testing  and  identifies  MOEs  and 
other  measures  for  critical  issues.  It  includes  developing  test  objectives,  establishing  a  test  approach, 
and  estimating  test  resource  requirements.  Detailed  planning  involves  developing  step-by-step 
procedures  to  be  followed  as  well  as  the  final  coordination  of  resource  requirements. 

10.6.1  Testing  COIs 

As  defined  in  Reference  (ah) ,  a  COI  is  a  key  operational  effectiveness  and/or  operational  suitability  issue 
(not  a  parameter,  objective,  or  threshold)  that  must  be  examined  in  OT&E  to  determine  the  systems 
capability  to  perform  its  mission.  A  COI  is  normally  phrased  as  a  question  that  must  be  answered  in 
order  to  properly  evaluate  operational  effectiveness  (e.g..  Will  the  system  detect  the  threat  in  a  combat 
environment  at  adequate  range  to  allow  successful  engagement?)  or  operational  suitability  (e.g..  Will 
the  system  be  safe  to  operate  in  a  combat  environment?).  A  COI  may  be  broken  down  into  a  set  of 
MOEs  and  MOSs.  The  latter  two  measures  in  turn  are  further  decomposed  into  supporting  MOPs.  See 
also  Figure  5-3. 

One  purpose  of  OT&E  is  to  resolve  COIs  about  the  system.  The  first  step  in  an  OT&E  program  is  to 
identify  these  critical  issues,  some  of  which  may  be  explicit  in  the  CDD.  Examples  can  be  found  in 
questions  such  as  the  following:  Flow  well  does  the  system  perform  a  particular  aspect  of  its  mission? 
Can  the  system  be  supported  logistically  in  the  field?  Other  issues  arise  from  questions  asked  about 
system  performance  or  how  the  system  will  affect  other  systems  with  which  it  must  operate.  Critical 
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issues  provide  focus  and  direction  for  the  OT.  Identifying  the  issues  is  analogous  to  the  first  step  in  the 
SE  process  that  is,  defining  the  problem. 

When  COIs  are  properly  addressed,  deficiencies  in  the  system  can  be  uncovered.  COIs  form  the  basis  for 
a  structured  technique  of  analysis  by  which  detailed  sub-objectives  or  MOEs  can  be  established.  During 
the  OT,  each  sub-objective  is  addressed  by  an  actual  test  measurement  (MOP).  After  these  issues  are 
identified,  the  evaluation  plans  and  test  design  are  developed  for  test  execution. 

10.6.2  Test  Realism 

Test  realism  for  OT&E  will  vary  directly  with  the  degree  of  system  maturity.  Efforts  early  in  the 
acquisition  program  should  focus  on  active  involvement  of  users  and  operationally  oriented 
environments.  Fidelity  of  the  combat  environment  should  peak  during  the  lOT&E  when  force-on-force 
testing  of  the  production-representative  system  is  conducted.  The  degree  of  success  in  replicating  a 
realistic  operational  environment  has  a  direct  impact  on  the  credibility  of  the  lOT&E  report.  Areas  of 
primary  concern  for  the  test  planner  can  be  derived  from  the  definition  of  OT&E,  as  stated  in  Section 
139  of  Reference  (b): 

•  A  field  test  includes  all  of  the  elements  normally  expected  to  be  encountered  in  the  operational 
arena,  such  as  appropriate  size  and  type  of  maneuver  terrain,  environmental  factors,  day/night 
operational  capability,  austere  living  conditions,  etc. 

•  Realistic  combat  should  be  replicated  using  appropriate  tactics  and  doctrine,  representative 
threat  forces  properly  trained  in  the  employment  of  threat  equipment,  free  play  responses  to 
test  stimulus,  stress,  dirty  battle  area  (fire,  smoke,  NBC,  electronic  countermeasures  (ECM)), 
wartime  tempo  to  operations,  real-time  casualty  assessment,  and  forces  requiring 
interoperability. 

•  Any  item  means  the  production  or  production-representative  configuration  of  the  system 
hardware  and  software  at  that  point  in  time,  including  appropriate  logistics  tail. 

•  Typical  military  users  are  obtained  by  taking  a  cross  section  of  adequately  trained  skill  levels  and 
ranks  of  the  intended  operational  force.  Selection  of  golden  crews  or  the  best-of-the-best  will 
not  provide  valid  test  data  reflecting  the  successes  or  problems  of  typical  units. 


10.6.3  Selection  of  a  Test  Concept 

An  important  step  in  the  development  of  an  OT&E  program  is  to  develop  an  overall  test  program 
concept.  Determinations  must  be  made  regarding  when  OT&E  will  be  conducted  during  systems 
development,  what  testing  is  to  be  done  on  production  equipment,  how  the  testing  will  be  evolutionary, 
and  what  testing  will  have  to  wait  until  all  system  capabilities  are  developed.  This  concept  can  best  be 
developed  by  considering  several  aspects  such  as  test  information  requirements,  system  availability  for 
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test  periods,  and  the  demonstration  of  system  capabilities.  The  test  concept  is  driven  by  the  acquisition 
strategy  and  is  a  roadmap  used  for  planning  T&E  events. 

10.7  Test  Execution 

An  OT  plan  is  only  as  good  as  the  execution  of  that  plan.  The  execution  is  the  essential  bridge  between 
test  planning  and  test  reporting.  The  test  is  executed  through  the  OTA  test  director's  efforts  and  the 
actions  of  the  test  team.  For  successful  execution  of  the  OT&E  plan,  the  test  director  must  direct  and 
control  the  test  resources  and  collect  the  data  required  for  presentation  to  the  decision  authority.  The 
test  director  must  prepare  for  testing,  activate  and  train  the  test  team,  develop  test  procedures  and 
operating  instructions,  control  data  management,  create  OT&E  plan  revisions,  and  manage  each  of  the 
test  trials.  The  test  directors  data  management  duties  will  encompass  collecting  raw  data,  creating  a 
data  status  matrix,  and  ensuring  data  quality  by  processing  and  reducing,  verifying,  filing,  storing, 
retrieving,  and  analyzing  collected  data.  Once  all  tests  have  been  completed  and  the  data  are  reduced 
and  analyzed,  the  results  must  be  reported. 

10.8  Test  Reporting 

The  Service  lOT&E  report  is  a  very  important  document.  It  must  communicate  the  results  of  completed 
tests  to  decision  authorities  in  a  timely,  factual,  concise,  comprehensive,  and  accurate  manner.  The 
report  must  present  a  balanced  view  of  the  weapon  systems  successes  and  problems  during  testing, 
illuminating  both  the  positive  aspects  and  system  deficiencies  discovered.  Analysis  and  evaluation  of 
test  data  may  be  in  one  report  (Air  Force,  Navy)  or  in  separate  documents  (Army,  Marine  Corps).  The 
Army's  independent  evaluation  report  (lER)  advises  the  decision  review  principals  and  the  MDA 
concerning  the  adequacy  of  testing,  the  systems  operational  effectiveness  and  suitability,  and 
survivability,  as  well  as  providing  recommendations  for  future  T&E  and  system  improvements. 

Types  of  reports  most  frequently  used  in  reporting  OT&E  results  include  status,  interim,  quick-look,  and 
final  reports.  The  status  report  gives  periodic  updates  (e.g.,  monthly,  quarterly)  and  reports  recent  test 
findings  (discrete  events  such  as  missile  firings).  The  interim  report  provides  a  summary  of  the 
cumulative  test  results  to  date  when  there  is  an  extended  period  of  testing.  The  quick-look  reports 
provide  preliminary  test  results,  are  usually  prepared  immediately  after  a  test  event  (within  7  days),  and 
have  been  used  to  support  program  decision  milestones.  The  final  T&E  report  (Air  Force,  Navy)  or  lER 
(Army,  Marine  Corps)  presents  the  conclusions  and  recommendations  including  all  supporting  data  and 
covering  the  entire  lOT&E  program.  The  Service  OTA  is  required  to  provide  reports  of  OT&E  results  in 
support  of  MS  B  and  MS  C,  in  addition  to  the  lOT&E  report  required  for  the  FRPDR. 

10.9  Summary 

The  purpose  of  OT&E  is  to  assess  operational  effectiveness  and  suitability  at  each  stage  in  the 
acquisition  process.  Operational  effectiveness  is  a  measure  of  the  contribution  of  the  system  to  mission 
accomplishment  under  actual  conditions  of  employment.  Operational  suitability  is  a  measure  of  the 
R&M  of  the  system;  the  effort  and  level  of  training  required  to  maintain,  support,  and  operate  the 
system;  and  any  unique  logistics  or  training  requirements  the  system  may  have.  The  OT&E  may  provide 
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information  on  tactics,  doctrine,  organization,  and  personnel  requirements  and  may  be  used  to  assist  in 
the  preparation  of  operating  and  maintenance  instructions  and  other  publications.  One  of  the  most 
important  aspects  of  OT&E  is  that  provides  an  independent  evaluation  of  the  degree  of  progress  made 
toward  satisfying  the  user's  requirements  during  the  system  development  process. 
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Test  &  Evaluation  Management  Guide 
Chapter  11  -  OT&E  to  Support  Decision  Reviews 

11.1  Introduction 

Mindful  of  principles  of  objectivity  and  impartial  evaluation,  OT&E  may  be  conducted  before  each 
decision  review  to  provide  the  decision  authority  with  the  latest  results  from  testing  of  COIs.  The 
philosophy  of  OT&E  has  been  related  to  three  terms-adequacy,  quality,  and  credibility: 

•  Adequacy  .  The  amount  of  data  and  realism  of  test  conditions  must  be  sufficient  to  support  the 
evaluation  of  the  COIs. 

•  Quality  .  The  test  planning,  control  of  test  events,  and  treatment  of  data  must  provide  clear  and 
accurate  test  reports. 

•  Credibility  .  The  conduct  of  the  test  and  data  handling  must  be  separated  from  external 
influence  and  personal  biases. 

OT  is  conducted  to  provide  information  to  support  executive-level  management  decisions  on  acquisition 
programs.  OT&E  is  accomplished  using  test  cycles  of  successive  actions  and  evaluations.  During  the  early 
stages  of  the  program,  the  process  is  informal  and  modified  as  necessary.  As  programs  mature, 
documentation  for  major  systems  and  those  designated  by  the  DOT&E  for  oversight  must  be  sent  to 
OSD  for  approval  before  the  testing  can  be  conducted  or  the  systems  can  be  cleared  to  proceed  beyond 
LRIP. 

11.2  OT&E  During  the  TD  Phase 

During  this  phase,  OT&E  may  be  conducted  on  brassboard  configurations,  experimental  prototypes,  or 
advanced  development  prototypes.  Dedicated  test  time  may  be  made  available  to  the  operational 
tester.  However,  the  OT&E  assessments  may  also  make  use  of  many  other  additional  data  sources. 
Examples  of  additional  sources  often  used  by  the  Army  during  this  phase  include  concept 
experimentation  program  tests,  innovative  testing,  FDT/E,  source  selection  tests,  market  investigations, 
warfighting  experiments,  and  user  participation  in  operational  feasibility  tests.  The  results  from  this 
testing,  analysis,  and  evaluation  are  documented  in  an  EOA  or  end-of-phase  OT&E  report.  These  data, 
along  with  the  mission  needs,  CONORS,  and  requirements  documentation  and  TEMP,  assist  in  the 
review  of  performance  for  the  next  decision  review. 

11.3  OT&E  During  the  EMD  Phase 

After  program  initiation,  integrated  DT/OT  or  an  OA  may  be  conducted  to  support  the  assessment  of 
prototype  development  and  a  decision  regarding  a  systems  readiness  to  move  into  the  development  of 
the  EDM.  In  all  cases,  appropriate  T&E  must  be  conducted  on  a  mature  system  configuration  (i.e.,  the 
EDM)  before  the  production  decision,  thereby  providing  data  for  identification  of  risk  before  more 
resources  are  committed.  As  appropriate,  LRIP  will  follow  this  phase  of  testing  to  verify  system-level 
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performance  capability  and  to  provide  insight  into  test  resources  needed  to  conduct  future 
interoperability,  live-fire,  or  operational  testing. 

EGAs  and  OAs  are  conducted  to  facilitate  identification  of  the  best  design,  indicate  the  risk  level  of 
performance  for  this  phase  of  the  development,  examine  operational  aspects  of  the  systems 
development,  and  estimate  potential  operational  effectiveness  and  suitability.  Additionally,  an  analysis 
of  the  planning  for  transition  from  development  to  production  is  initiated.  OAs  and  EGAs  supporting 
decision  reviews  are  intended  to: 

•  Assess  the  potential  of  the  new  system  in  relation  to  existing  capabilities. 

•  Assess  system  effectiveness  and  suitability  so  that  affordability  can  be  evaluated  for  program 
cost  versus  military  utility. 

•  Assess  the  adequacy  of  the  employment  concept;  supportability;  organizational  doctrine, 
tactical,  and  training  requirements;  and  related  critical  issues. 

•  Estimate  the  need  for  the  selected  systems  in  consideration  of  the  threat  and  system 
alternatives  based  on  military  utility. 

•  Assess  the  validity  of  the  operational  concept.  List  the  key  risk  areas  and  CGIs  that  need  to  be 
resolved  before  construction  of  EDMs  is  initiated. 

•  Assess  the  need  during  LRIP  of  long-lead  hardware  to  support  IGT&E  prior  to  the  FRP  decision. 

•  Provide  data  to  support  test  planning  for  this  phase. 

The  GAs  during  the  system  capability  demonstration  are  conducted  on  EDMs.  These  operational 
evaluations  estimate  operational  effectiveness  and  suitability  and  provide  data  on  whether  the  system 
meets  minimum  operational  thresholds. 

11.4  OT&E  During  the  P&D  Phase 

Just  before  the  FRP  decision,  the  Services  dedicated  GT&E  is  conducted  on  production  equipment  that 
has  been  formally  certified  by  the  PM  during  the  GTRR  as  being  ready  for  the  final  GT&E.  GSD  oversight 
programs  also  consider  the  DASD(DT&E)s  AGTR.  The  dedicated  IGT&E  is  conducted  in  a  test 
environment  as  operationally  realistic  as  possible.  GSD  oversight  programs  use  the  IGT&E  plan  approved 
by  DGT&E. 

The  IGT&E  conducted  is  characterized  by  testing  performed  by  user  organizations  in  a  field  exercise  to 
examine  the  organization  and  doctrine,  ILS,  threat,  communications,  C2,  and  tactics  associated  with  the 
operational  employment  of  the  unit  during  tactical  operations.  This  includes  estimates  that: 

•  Assess  operational  effectiveness  and  operational  suitability. 

•  Assess  the  survivability  of  the  system. 
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•  Assess  reliability,  maintainability,  and  plans  for  ILS. 

•  Evaluate  manpower,  personnel,  training,  and  safety  requirements. 

•  Validate  organizational  and  employment  concepts. 

•  Determine  training  and  logistics  deficiencies. 

•  Assess  the  systems  readiness  to  enter  FRP. 

11.5  OT&E  During  the  O&S  Phase 

After  the  FRP  decision  and  deployment,  the  emphasis  shifts  toward  procuring  production  quantities, 
repairing  hardware  deficiencies,  managing  changes,  and  phasing  in  full  logistics  support.  During  initial 
deployment  of  the  system,  the  OT&E  agency  and/or  the  user  may  perform  FOT&E  to  refine  the 
effectiveness  and  suitability  estimates  made  during  earlier  OT&E,  assess  performance  not  evaluated 
during  lOT&E,  evaluate  new  tactics  and  doctrine,  and  assess  the  impacts  of  system  modifications  or 
upgrades. 

The  FOT&E  is  performed  with  production  articles  in  operational  organizations.  It  is  frequently  funded 
with  operations  and  maintenance  (O&M)  funds.  FOT&E  is  normally  funded  from  the  appropriation  most 
suitable  to  the  life  cycle  phase  of  the  system  involved.  The  first  FOT&E  conducted  during  this  phase  may 
be  used  to: 

•  Ensure  that  the  production  system  performs  as  well  as  reported  at  the  MS  C  review. 

•  Demonstrate  expected  performance  and  reliability  improvements. 

•  Ensure  the  correction  of  previously  identified  deficiencies. 

•  Evaluate  performance  not  tested  during  lOT&E. 

Additional  objectives  of  FOT&E  are  selected  to  validate  the  operational  effectiveness  and  suitability  of  a 
modified  system  during  an  OA  of  the  system  in  new  environments.  The  FOT&E  may  look  at  different 
platform  applications,  new  tactical  applications,  or  the  impact  of  new  threats. 

The  testing  objectives  to  evaluate  post-production  logistics  readiness  and  support  are  to: 

•  Assess  the  logistics  readiness  and  sustainability. 

•  Evaluate  the  weapons  system  support  objectives/metrics. 

•  Assess  the  implementation  of  logistics  support  planning. 

•  Evaluate  the  capability  of  the  logistics  support  activities. 
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11.6  Summary 

OT&E  is  that  T&E  (OAs,  lOT&E,  or  FOT&E)  conducted  by  an  independent  OT&E  agency  to  provide 
feedback  on  system  design  and  the  systems  potential  to  be  operationally  effective  and  operationally 
suitable.  OT&E  events  in  each  phase  of  development  will  identify  needed  modifications;  provide 
information  on  tactics,  doctrine,  organizations,  and  personnel  requirements;  and  evaluate  the  systems 
logistics  supportability.  The  acquisition  program  structure  should  include  feedback  from  OAs  or 
evaluations  at  critical  decision  points/technical  reviews  beginning  early  in  and  continuing  throughout 
the  systems  life  cycle. 
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Test  &  Evaluation  Management  Guide 
Chapter  12 -TEMP 

12.1  Introduction 

Guidance  contained  in  References  and  (]}  stipulates  that  a  TEMP  shall  be  used  for  all  ACAT  I  and  lA  or 
OSD-designated  oversight  acquisition  programs. 

For  effective  engineering  development  and  decision-making  processes,  an  early  T&E  strategy  in  the  TD 
phase  must  be  evolved  into  an  overall  integrating  master  plan  detailing  the  collection  and  evaluation  of 
test  data  on  required  performance  parameters.  Less  than  ACAT  I  programs  are  encouraged  to  tailor  their 
T&E  strategy  using  the  TEMP  format  provided  in  Chapter  9  of  Reference  (j)  as  a  guide. 

The  TEMP  relates  program  schedule,  test  management  strategy  and  structure,  and  required  resources 
to  COIs,  CTPs,  minimum  performance  values  (thresholds),  acquisition  strategy,  and  milestone  decision 
points.  The  TEMP  serves  as  the  overarching  document  for  managing  a  T&E  program.  PMs  develop  the 
initial  TEMP  at  MS  B.  Prior  to  each  subsequent  Defense  Acquisition  System  milestone,  the  PMs  must 
submit  an  updated  TEMP.  The  TEMP  should  include  sufficient  detail  to  support  development  of  other 
test-related  documents. 

12.2  TEMP  Development 

The  T&E  strategy  is  highly  contingent  on  early  system  concepts  that  are  deemed  appropriate  for 
satisfying  user  requirements.  As  outlined  in  Reference  (h) ,  the  priority  for  selecting  a  solution  to  meet  a 
capability  shortfall  is  as  follows: 

1.  A  non-materiel  solution,  such  as  changes  to  doctrine,  organization,  training,  leadership  and 
education,  personnel,  and  facilities. 

2.  A  materiel  alternative,  chosen  according  to  the  following  priority  sequence: 

a.  The  procurement  or  modification  of  commercially  available  products,  services,  and 
technologies  from  domestic  or  international  sources,  or  the  development  of  dual-use 
technologies. 

b.  The  additional  production  or  modification  of  previously  developed  U.S.  and/or  allied 
military  systems  or  equipment. 

c.  A  cooperative  development  program  with  one  or  more  allied  nations. 

d.  A  new,  joint,  DoD  Component  or  Government  agency  development  program. 

e.  A  new  DoD  Service-unique  development  program. 

Development  of  the  system  TEMP  begins  during  the  MSA  effort  with  the  development  of  an  early  T&E 
strategy  for  the  proposed  materiel  solution.  This  T&E  strategy  evolves  as  the  system  is  better  defined  in 
the  TD  phase  and  T&E  events  are  consolidated  in  the  TEMP.  Effective  management  of  the  various  test 
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processes  is  the  primary  function  of  a  PMO  T&E  WIPT.  The  quality  of  the  test  program  may  directly 
reflect  the  level  of  effort  expended  in  its  development  and  execution.  This  effort  varies  in  direct 
relationship  to  the  management  imposed  by  the  PM  and,  to  some  extent,  by  the  system  engineer.  The 
PM  should  begin  the  TEMP  development  process  as  early  as  possible.  It  is  strongly  recommended  that 
the  PM  have  in-house  TEMP  personnel  with  expertise  in  T&E  WIPTs  and  TEMP  development.  The  T&E 
WIPT  will  make  many  decisions  that  impact  the  program.  It  is  important  that  the  PM  have  experienced 
test  staff  to  conduct  the  T&E  WIPT,  develop  the  TEMP,  and  provide  the  PM  with  advice  and  guidance. 

12.2.1  PMO  Responsibilities 

The  PMO  is  the  focal  point  of  the  development,  review,  and  submittal  process  for  the  program  TEMP. 
The  DoD  acquisition  process  requires  a  TEMP  as  one  of  the  primary  management  strategy  documents 
supporting  the  decision  to  begin  development  efforts.  However,  developing  a  TEMP  at  this  point  in  time 
can  be  difficult  as  some  Services  do  not  formulate  or  staff  a  PMO  until  formal  program  initiation  at  MS  B. 
An  additional  complicating  factor  is  the  nebulous  condition  of  other  program  source  documents  (CDD, 
technical  management  plan,  acquisition  strategy,  STA,  logistics  support  plan,  etc.)  that  are  also  in  early 
stages  of  development/updating  for  the  milestone  review. 

Because  the  TEMP  must  conform  to  the  acquisition  strategy  and  other  program  management 
documents,  it  frequently  lags  in  the  development  process  and  does  not  receive  the  attention  needed 
from  PMO  or  matrix  support  personnel.  PMO  emphasis  on  early  formulation  of  the  test  planning  teams 
(T&E  WIPT)  is  critical  to  the  successful  development  of  the  program  TEMP.  These  teams  should  consist 
of  the  requisite  players  so  a  comprehensive  and  integrated  strategy  compatible  with  other  engineering 
and  decision-making  processes  is  developed.  An  early  start  in  getting  Service-level  concurrence  is 
important  so  the  milestone  decision  document-submission  schedule  can  be  supported  with  the  draft 
and  final  versions  of  the  TEMP.  Subsequent  updates  do  not  become  easier,  as  each  acquisition  phase 
brings  new  planning,  coordination,  and  testing  requirements. 

12.2.2  T&E  Planning 

Developing  an  overall  strategy  provides  the  framework  for  incorporating  phase-oriented  T&E  activities 
that  will  facilitate  the  acquisition  process.  The  T&E  strategy  should  be  consistent  with  the  program 
acquisition  strategy,  identifying  requirements  for  contractor  and  Government  DT&E,  interactions 
between  DT  and  OT,  and  provisions  for  the  separate  lOT&E. 

An  evolutionary  acquisition  strategy  is  the  preferred  strategy  for  the  rapid  acquisition  of  mature 
technology  for  the  user  and  will  generally  include  an  incremental  development  approach  using  low-risk 
technologies  that  should  reduce  the  intensity  and  duration  of  the  T&E  program.  It  does,  however, 
include  a  requirement  for  post-production  test  activities  as  the  system  is  modified  to  accommodate 
previously  unknown  new  technologies,  new  threats,  or  other  performance  enhancements. 

A  Grand  Design  or  single-step  acquisition  strategy  incorporates  all  the  latest  technologies  in  the  final 
production  configuration  and  is  generally  a  higher  risk  approach.  As  the  contractor  works  on  maturing 
emerging  technologies,  the  T&E  workload  increases  in  direct  proportion  to  the  technology  risk  and 
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difficulty  in  fixing  problems.  There  is  a  much  higher  potential  for  extended  schedules  with  iterative  test- 
fix-test  cycles . 

12.2.3  General  T&E  Planning  Considerations 

A  variety  of  reviews  have  identified  have  identified  several  factors  pertaining  to  DoD  systems  that 
should  be  considered  during  test  planning.  These  considerations,  along  with  a  summary  discussion,  are 
provided  below . 

12.2.3.1  Effects  of  Test  Requirements  on  System  Acquisition 

The  acquisition  strategy  for  the  system  should  allow  sufficient  time  and  flexibility  between  the  end  of 
the  EMD  phase  and  the  start  of  LRIP  to  adjust  for  modifications  and  associated  testing.  It  should  ensure 
that  sufficient  dollars  are  available  not  only  to  conduct  T&E  but  also  to  allow  for  additional  T&E  that  is 
always  required  due  to  failure,  design  changes,  etc.  The  acquisition  strategy  should  be  evaluated  relative 
to  constraints  imposed  by: 

•  The  level  of  system  testing  at  various  stages  of  the  RDT&E  cycle. 

•  The  number  of  test  items  available  and  the  schedule  interface  with  other  systems  needed  in  the 
tests,  such  as  aircraft,  electronics,  etc. 

•  The  support  required  to  assist  in  preparing  for  and  conducting  tests  and  analyzing  the  test 
results. 

•  The  number  of  systems  being  evaluated  to  minimize  the  so-called  T&E  gap  caused  by  lack  of 
hardware  during  the  test  phase. 

12.2.3.2  Test  Requirements  and  Restrictions 

Tests  should: 


•  Have  specific  objectives. 

•  List,  in  advance,  actions  to  be  taken  as  a  consequence  of  the  test  results. 

•  Be  instrumented  to  permit  diagnosis  of  the  cause  of  lack  of  performance  including  random, 
design-induced  wear-out  and  operator  error  failure. 

•  Not  be  repeated  if  failures  occur  without  first  conducting  a  detailed  analysis  of  the  failure. 

•  Include  test  rehearsals  for  each  new  phase  of  testing. 

12.2.3.3  Use  of  Government  Test  Facilities 

Programs  should  use  DoD  Government  T&E  capabilities  and  invest  in  Government  T&E  infrastructure 
unless  an  exception  can  be  justified  as  cost-effective  to  the  Government. 
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12.2.3.4  Trouble  Indicators 

Programs  should  establish  an  early  detection  scheme/measurement  process  to  identify  program 
problems  and  issues  in  development  as  early  as  possible  and  correct  them.  As  one  of  the  DoD  SE 
technical  management  processes,  outputs  from  the  technical  assessment  process  can  help  to  provide 
those  early  insights.  The  technical  assessment  process  consists  of  EVM,  TPM,  and  technical  reviews  and 
audits.  When  a  program  has  not  established  such  a  process,  indicators  will  show  up  during  testing.  Some 
of  these  indicators  include: 

•  Any  repetitive  failure. 

•  A  revision  of  schedule  or  incremental  funding  that  exceeds  the  original  plan. 

•  Any  relaxation  of  the  basic  requirements  such  as  lower  performance. 

•  Cost  overruns. 

12.2.3.5  RAM 

RAM  have  a  direct  impact  on  both  operational  capability  and  total  ownership  cost  (TOC)  and  therefore 
are  important  considerations  for  the  Warfighter.  Achieving  required  levels  of  RAM  for  a  system  is 
important  for  many  reasons,  some  of  which  are  identified  below: 

•  Improved  readiness.  Poor  reliability  or  maintainability  causes  readiness  to  fall  below  needed 
levels  or  increases  the  cost  of  achieving  the  needed  levels. 

•  Improved  safety.  Inadequate  reliability  of  components  deemed  critical  safety  items  (CSIs)  may 
directly  jeopardize  the  safety  of  the  user(s)  of  that  components  system  and  result  in  a  loss  of 
life.  The  ability  to  complete  a  mission  safely  is  directly  related  to  the  reliability  of  the  CSIs. 

•  Improved  mission  success.  Inadequate  reliability  of  equipment  directly  jeopardizes  mission 
success  and  may  result  in  undesirable  repetition  of  the  mission. 

•  Reduced  TOC.  The  concept  of  TOC  captures  the  true  cost  of  design,  development,  ownership, 
and  support  of  DoD  weapons  systems. 

The  T&E  strategy  and  the  TEMP  must  specify  how  reliability  will  be  tested  and  evaluated  during  the 
associated  acquisition  phase.  RGCs  will  be  developed  to  reflect  the  reliability  growth  strategy  and  will  be 
employed  to  plan,  illustrate,  and  report  reliability  growth.  An  RGC  will  be  included  in  the  SEP  at  MS  A 
and  updated  in  the  TEMP  beginning  at  MS  B.  The  RGC  will  be  stated  in  a  series  of  intermediate  goals  and 
tracked  through  fully  integrated,  system-level  T&E  events  until  the  reliability  threshold  is  achieved,  see 
Reference  (o) .  PMs  and  the  PMO  are  responsible  for  reporting  progress  to  plan  for  RAM  at  the  Defense 
Acquisition  Executive  Summary  (DAES)  reviews. 
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12.2.3.6  Requirement  for  Test  Rehearsals 

Test  rehearsals  should  be  conducted  for  each  new  phase  of  testing.  They  provide  opportunities  to 
validate  operator  training,  safety  issues,  data  collection  systems,  implementation  of  scenarios,  and 
management  of  post-test  data  analysis  . 

12.2.4  Test  Scheduling  Criteria 

Specific  issues  associated  with  test  scheduling  are  discussed  in  the  following  sections.  Criteria  for  who 
and  what  constitutes  grounds  for  stopping  or  delaying  the  test(s)  must  be  established  as  part  of  the  test 
planning  process.  Additionally,  test  schedules  should  have  some  margin  for  retest  events  and  possible 
associated  development  rework  to  fix  discovered  problems. 

As  part  of  integrated  testing,  some  OT&E  is  interwoven  into  early  DT&E  for  maximizing  the  efficient  use 
of  test  articles  and  test  schedules.  However,  OT&E  must  remain  a  distinct  thread  of  activity  that  does 
not  lose  its  identity  in  the  tapestry  of  test  events. 

12.2.4.1  Building  Block  Test  Scheduling 

A  set  of  tests  to  demonstrate  feasibility  prior  to  testing  the  system-level  EDMs  should  be  used.  This 
approach  will  allow  early  testing  of  high-technical-risk  items,  and  subsequent  tests  can  be  incorporated 
into  the  hardware  as  the  system  concept  has  been  demonstrated  as  feasible. 

12.2.4.2  Component  and  Subsystem  Test  Plans 

Ensure  that  a  viable  component  and  subsystem  test  plan  is  used.  Studies  show  that  almost  all 
component  failures  will  be  the  kind  that  cannot  be  easily  detected  or  prevented  in  full-system  testing. 
System  failure  must  be  detected  and  fixed  at  the  lowest  level  possible  in  the  system  WBS,  ideally  at  the 
component/subsystem  stage.  Detecting  and  correcting  failure  only  at  the  system  level  and  during  high- 
visibility  OT  results  in  program  cost  and  schedule  overruns  . 

12.2.4.3  Schedule  lOT&E  to  Include  System  Interfaces  with  Other  Systems 

Whenever  possible,  the  lOT&E/FOT&E  of  a  weapon  system  should  be  planned  to  include  other  systems 
that  must  have  a  technical  interface  with  the  new  system.  For  example,  missiles  should  be  tested  on 
most  of  the  platforms  for  which  they  are  programmed  . 

12.2.4.4  Test  Resources 

Planning  for  test  resources  is  driven  by  the  sequence  and  intensity  of  DT  and  OT  events.  Resource 
coordination  is  an  equally  arduous  task,  which  frequently  has  lead  times  equal  to  major  program 
development  activities.  The  program  T&E  strategy  should  include  an  overshadowing  evaluation  plan, 
outlining  methodologies,  models,  simulations,  and  test  data  required  at  periodic  decision  points. 
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12.3  TEMP  Requirements 

The  TEMP  should  address  critical  human  issues  to  provide  data  to  validate  the  results  of  human  factors 
engineering  analyses  and  require  identification  of  mission  critical  operational  and  maintenance  tasks. 

As  early  as  practicable,  developers  and  test  agencies  will  assess  survivability  and  validate  critical 
survivability  characteristics  at  as  high  a  system  level  as  possible.  The  TEMP  will  identify  the  means  by 
which  the  survivability  objective  will  be  validated. 

Field  engineering  test  facilities  and  testing  in  the  intended  operational  environments  are  required  to 
verify  electric  or  electronic  systems  predicted  performance,  establish  confidence  in  electromagnetic 
compatibility  design  based  on  standards  and  specifications,  and  validate  electromagnetic  compatibility 
analysis  methodology. 

The  TEMP  should: 

•  Address  health  hazard  and  safety  critical  issues  to  provide  data  to  validate  the  results  of  system 
safety  analyses. 

•  Directly  support  the  development  of  more  detailed  planning  and  resource  documents  needed  to 
execute  the  actual  test  events  and  subsequent  evaluations. 

•  Provide  a  roadmap  for  integrated  simulation,  test,  and  evaluation  plans,  schedules,  and 
resource  requirements  necessary  to  accomplish  the  T&E  program.  T&E  planning  should  address 
MOEs/MOSs  with  appropriate  quantitative  criteria,  test  event  or  scenario  description,  resource 
requirements,  and  test  limitations.  Test  planning,  at  a  minimum,  must  address  all  system 
components  that  are  critical  to  the  achievement  and  demonstration  of  contract  technical 
performance  specifications  and  threshold  values  specified  in  the  CPD. 

•  Identify  the  Lead  Developmental  Tester  and  the  Lead  Government  DT&E  Organization  ( 
Reference  (w) ) . 

12.4  TEMP  Format 

The  recommended  TEMP  format,  as  shown  in  Figure  12-1,  is  for  all  programs,  regardless  of  acquisition 
category,  and  is  available  in  Chapter  9  of  Reference  (j).  The  TEMP  is  intended  to  be  a  summary 
document  outlining  DT&E,  OT&E,  and  LFT&E  management  responsibilities  across  all  phases  of  the 
acquisition  process.  When  the  development  is  a  multi-Service  or  joint  acquisition  program,  one 
integrated  TEMP  is  developed  with  Service  annexes,  as  required.  A  Capstone  TEMP  may  not  be 
appropriate  for  a  single  major  weapon  platform  but  could  be  used  to  encompass  testing  of  a  collection 
of  individual  systems,  each  with  its  own  annex.  A  program  TEMP  is  updated  at  milestones,  upon 
program  baseline  breach,  and  for  other  significant  program  changes.  Updates  may  consist  of  page 
changes  and  are  no  longer  required  when  a  program  has  no  further  development  activities. 
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2.2  Common  T&E  Database  Requirements 

4.1.3  Test  Support  Equipment 
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2.5  Integrated  Test  Program  Schedule 

4.1.6  Operational  Force  Test  Support 

Figure  2.1  -  Integrated  Test  Program  Schedule 

Part  III  -  Test  and  Evaluation  Strategy 
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beds 

3.1  T&E  Strategy 

3.2  Evaluation  Framework 

4.1.8  Joint  Operational  Test 

Environment 

Figure  3.1  -  Top-Level  Evaluation  Framework 
Matrix 

4.1.9  Special  Requirements 

3.3  Developmental  Evaluation  Approach 

4.2  Federal,  State,  Local  Requirements 

3.3.1  Mission-Oriented  Approach 

4.3  Manpower/Personnel  Training 
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3.3.3  Modeling  and  Simulation 

Table  4.1  Resource  Summary  Matrix 

3.3.4  Test  Limitations 

Appendix  A  -  Bibliography 

3.4  Live  Fire  Evaluation  Approach 

Appendix  B  -  Acronyms 
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Appendix  C  -  Points  of  Contact 
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3.4.3  Test  Limitations 

Figure  12-1  TEMP  Format 

The  TEMP  is  a  living  document  that  must  address  changes  to  critical  issues  associated  with  an 
acquisition  program.  Major  changes  in  program  requirements,  schedule,  or  funding  usually  result  in  a 
change  in  the  test  program.  Thus,  the  TEMP  must  be  reviewed  and  updated  on  program  change,  on 
baseline  breach,  and  before  each  milestone  decision  to  ensure  that  T&E  requirements  are  current.  As 
the  primary  document  used  in  the  OSD  review  and  milestone  decision  process  to  assess  the  adequacy  of 
planned  T&E,  the  TEMP  must  be  of  sufficient  scope  and  content  to  explain  the  entire  T&E  program. 

Each  TEMP  submitted  to  OSD  should  be  an  integrated  document,  detailed  enough  to  show  the  rationale 
for  the  type,  amount,  and  schedules  of  the  testing  planned.  It  must  clearly  relate  the  T&E  effort  to 
technical  risks,  operational  issues  and  concepts,  system  performance,  RAM  growth,  logistics  objectives 
and  requirements,  and  major  decision  points.  The  TEMP  should  summarize  the  testing  accomplished  to 
date  and  explain  the  relationship  of  the  various  models  and  simulations,  subsystem  tests,  integrated 
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system  developmental  tests,  and  initial  operational  tests  that,  when  analyzed  in  combination,  provide 
confidence  in  the  systems  readiness  to  proceed  into  the  next  acquisition  phase. 

The  TEMP  must  address  the  T&E  to  be  accomplished  in  each  program  phase,  with  the  next  phase 
addressed  in  the  most  detail.  The  TEMP  is  also  used  as  a  collaboration  document  to  outline  each  test 
and  support  organizations  role  in  the  T&E  program  and  identify  major  test  facilities  and  resources. 

The  TEMP  should  be  aligned,  wherever  appropriate,  with  the  SEP  that  is  used  on  the  program  and 
guides  overall  technical  development  activities. 

The  objective  of  the  OSD  TEMP  review  process  is  to  ensure  successful  T&E  programs  that  will  support 
decisions  to  commit  resources  at  major  milestones.  Some  of  the  T&E  questions  considered  during  the 
TEMP  review  process  include  the  following: 

•  Are  DT&E,  OT&E,  and  LFT&E  initiated  early  to  assess  performance,  identify  risks,  and  estimate 
operational  potential? 

•  Are  critical  issues,  test  directives,  and  evaluation  criteria  related  to  mission  need  and 
operational  requirements  established  well  before  testing  begins? 

•  Are  provisions  made  for  collecting  sufficient  test  data  with  appropriate  test  instrumentation  to 
minimize  subjective  judgment? 

•  Is  lOT&E  conducted  by  an  organization  independent  of  the  developer  and  user? 

•  Do  the  test  methodology  and  instrumentation  provide  a  mature  and  flexible  network  of 
resources  that  stresses  (as  early  as  possible)  the  weapon  system  in  a  variety  of  realistic 
environments? 

•  Is  an  integrated  approach  to  DT,  OT,  and  LET  being  taken  where  appropriate? 

12.5  Summary 

The  PMO  is  directly  responsible  for  the  content  and  quality  of  the  test  strategy  and  associated  planning 
activities  that  are  documented  in  the  TEMP  and  other  plans.  The  TEMP,  as  an  integrated  summary 
management  tool,  requires  a  commitment  of  staff-hours  and  PM  guidance  for  successful  completion. 
The  interactions  of  the  various  T&E  players  and  support  agencies  in  the  T&E  WIPT  must  be  woven  into 
the  fabric  of  the  total  system  acquisition  strategy  and  an  integrated  testing  approach.  Cost  and  schedule 
implications  must  be  negotiated  to  ensure  a  viable  T&E  program  that  provides  timely  and  accurate  data 
to  the  engineering  and  management  decision  makers. 
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Test  &  Evaluation  Management  Guide 
Chapter  13  -  Test  Resources 

13.1  Introduction 

This  chapter  describes  the  various  types  of  resources  available  for  testing,  explains  test  resource 
planning  in  the  Services,  and  discusses  the  ways  in  which  test  resources  are  funded. 

Test  resources  is  a  collective  term  that  encompasses  elements  necessary  to  plan,  conduct,  collect,  and 
analyze  data  from  a  test  event  or  program.  These  elements  include  items  such  as  funding  (to  develop 
new  resources  or  use  existing  ones),  manpower  for  test  conduct  and  support,  test  articles,  models, 
simulations,  threat  simulators,  surrogates,  replicas,  test  beds,  special  instrumentation,  test  sites,  targets, 
tracking  and  data  acquisition  instrumentation,  equipment  (for  data  reduction,  communications, 
meteorology,  utilities,  photography,  calibration,  security,  recovery,  maintenance,  and  repair),  frequency 
management  and  control,  and  base/facility  support  services.  As  a  general  principle,  test  planning  and 
conduct  should  take  full  advantage  of  existing  investments  in  DoD  resources.  Programs  must  use  DoD 
Government  T&E  capabilities  and  invest  in  Government  T&E  infrastructure  unless  an  exception  can  be 
justified  as  cost-effective  to  the  Government. 

Key  DoD  test  resources  are  in  great  demand  by  competing  acquisition  programs.  Special,  unique,  or  one- 
of-a-kind  test  resources  must  often  be  developed  specifically  for  the  test  program.  Therefore,  it  is 
imperative  that  the  requirements  for  these  test  resources  be  identified  early  in  the  acquisition  process 
so  adequate  funding  can  be  allotted  for  development  of  the  test  resources  and  they  will  be  available 
when  the  test  is  scheduled. 

13.2  Obtaining  Test  Resources 

13.2.1  Test  Resources  and  Instrumentation 

As  early  as  possible  but  not  later  than  program  start,  the  test  facilities  and  instrumentation 
requirements  to  conduct  program  T&E  should  be  identified  and  a  tentative  schedule  of  test  activities 
should  be  prepared.  This  information  is  developed  in  the  early  T&E  strategy  and  expanded  in  the  TEMP 
and  Service-specific  test  resource  documentation.  PMs  must  conduct  a  CBA  when  unable  to  use  DoD 
Government  T&E  capabilities  or  unable  to  invest  in  Government  T&E  infrastructure  and  must  document 
the  assumptions  and  results  of  the  CBA  in  an  approved  TEMP. 

13.2.2  MOT&E 

MOT&E  should  be  considered  for  weapon  systems  requiring  new  operational  concepts  involving  more 
than  one  Service.  An  analysis  should  be  conducted  before  the  low-rate  production  decision  about  the 
impact  of  demonstration  on  time  and  resources  needed  to  execute  the  multi-Service  tests. 
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13.2.3  Military  Construction  Program  Facilities 

Some  programs  cannot  be  tested  without  military  construction  program  facilities.  Constructing  these 
facilities  will  require  long  lead  times  and  Military  Construction  funding;  therefore,  early  planning  must 
be  done  to  ensure  that  the  facilities  will  be  available  and  operational  when  required.  In  accordance  with 
DoDD  4270.5  (Reference  (am)),  construction  projects  should  normally  be  justified  and  funded  through 
the  planning,  programming,  and  budgeting  process.  Projects  shall  be  considered  for  funding  under 
authorities  available  to  the  Heads  of  the  DoD  Components  before  being  considered  for  funding  under 
the  authority  of  the  Secretary  of  Defense. 

13.2.4  Test  Sample  Size  Considerations 

The  primary  basis  for  the  test  sample  size  is  usually  based  on  one  or  more  of  the  following: 

•  Coverage  of  test  objectives,  particularly  KPPs. 

•  Significance  of  test  results  at  some  specified  statistical  confidence  level. 

•  Delta  the  difference  that  needs  to  be  detected. 

•  Availability  of  test  vehicles,  software,  support  items,  etc. 

•  Support  resources  or  facilities  available. 

13.2.5  Test  Termination 

Testers  should  not  hesitate  to  terminate  a  test  before  its  completion  if  it  becomes  clear  that  the  main 
objective  of  the  test  has  already  been  achieved  or  is  unachievable  (due  to  hardware  failure, 
unavailability  of  resources,  etc.)  or  if  additional  samples  will  not  change  the  outcome  and  conclusions  of 
the  test. 

13.2.6  Budgeting  for  Testing 

The  Acquisition  Strategy,  TEMP,  and  budgeting  documents  should  be  reviewed  regularly  to  ensure  that 
adequate  testing  funds  are  identified.  These  documents  need  careful  scrutiny  to  ensure  that  adequate 
contingency  funds  exist  to  cover  correction  of  difficulties  at  a  level  that  matches  industry/Government 
experience  on  the  contract.  Retesting  to  correct  deficiencies  found  during  testing  without  sufficient 
funding  for  proper  correction  results  in  Band-Aid  approaches,  which  require  more  expensive  corrections 
later. 

13.2.6.1  Cost  Analysis  Requirements  Description  (CARD) 

For  ACAT  I  and  ACAT  lA  programs,  the  CARD  is  used  to  formally  describe  the  acquisition  program  for 
purposes  of  preparing  both  the  DoD  Component  cost  estimate  and  the  cost  assessment  independent 
cost  estimate.  Reference  (d)  specifies  that  for  MDAPs,  the  CARD  will  be  provided  in  support  of  major 
milestone  decision  points  (MS  B,  MS  C,  or  the  FRPDR).  In  addition,  for  MAIS  programs,  the  CARD  is 
prepared  whenever  an  economic  analysis  is  required.  For  other  acquisition  programs,  the  preparation  of 
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a  CARD,  or  an  abbreviated  CARD-like  document  with  appropriate  tailoring,  is  strongly  encouraged  to 
provide  a  written  program  description  suitable  to  support  a  credible  LCC  estimate. 

The  CARD  is  prepared  by  the  program  office  and  approved  by  the  DoD  Component  PEO.  For  joint 
programs,  the  CARD  includes  the  common  program  agreed  to  by  all  participating  DoD  Components  as 
well  as  all  unique  program  requirements  of  the  participating  DoD  Components.  Chapter  1  of  DoD 
5000.4-IVI  (Reference  (an))  provides  further  guidelines  for  CARD  content.  Common  source  documents  for 
the  CARD  include  the  following: 

•  TRA. 

•  Capability  needs  documents  (i.e.,  ICD/CDD/CPD). 

•  Acquisition  Strategy. 

•  Life-cycle  sustainment  plan  (LCSP)  (part  of  the  Acquisition  Strategy). 

•  TEMP. 

•  Manpower  estimate. 

•  SEP. 

13.2.6.2  Test  and  Evaluation  Budget  Exhibit  (T&E-l) 

Optimizing,  sustaining,  and  improving  capabilities  directly  relates  to  a  clear  understanding  of  the  costs 
that  make  up  and  flow  through  these  test  capabilities,  as  well  as  the  type,  mix,  knowledge,  and  skills  of 
the  workforce,  section  139b  of  Reference  (b)  requires  the  DASD(DT&E)  to  periodically  review  of  the 
organizations  and  capabilities  of  the  Military  Departments  with  respect  to  DT&E  and  identify  needed 
changes  or  improvements  to  those  capabilities.  The  DASD(DT&E)  established  a  process  for  evaluating 
the  required  funding  for  programs  under  DT&E  oversight.  This  process  identifies  the  resourcing 
requirements  for  MDAPs  as  well  as  some  special  interest  programs.  An  updated  and  relevant  resource 
section  of  the  TEMP  is  required  and  can  be  evaluated  in  several  ways: 

•  The  total  program  cost  can  be  evaluated  by  reviewing  the  SARs  and  the  RDT&E  Budget  Item 
Justification,  Exhibit  R-2.  These  two  documents  allow  identification  of  total  program  costs  by 
calculating  the  values  in  the  Prior  Totals,  Future  Years  Defense  Program  (FYDP),  and  To 
Complete  sections. 

•  Starting  with  the  FY  2012  budget  cycle,  the  T&E-l  Budget  Exhibit  will  capture  numbers  for  prior 
years  and  cost  to  complete,  providing  an  additional  verification  of  total  program  costs. 

•  The  T&E-l  data  and  Part  IV  (Resource  Summary)  of  the  TEMP  can  be  compared.  Both 
documents  break  down  the  FYDP  costs  by  program  element  (PE).  Each  PE  identifies  funds 
allocated  to  different  types  of  testing  (DT&E,  lOT&E,  and  LFT&E)  and  identifies  the  appropriation 
type  (RDT&E,  procurement,  etc.). 
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Chapter  19  of  Volume  2B  of  DoD  7000. 14-R  (Reference  (ao))  provides  guidance  on  completing  Exhibit 
T&E  1. 

13.2.7  Test  Article  Planning 

Test  article  planning  should: 

•  Ensure  that  the  whole  system,  including  system  user  personnel,  is  tested. 

•  Realistically  test  the  complete  system,  including  hardware,  software,  people,  and  all  interfaces. 

•  Get  the  user  involved  from  the  start  and  understand  user  limitations. 

•  Ensure  that  sufficient  time  and  test  articles  will  be  available  (when  technology  is  stressed,  more 
test  articles  and  time  may  be  required). 

•  Test  parts,  then  subsystems,  and  finally  systems  to  ensure  that  they  work  as  prescribed  before 
incorporating  them  into  the  next  higher  assembly. 

•  Plan  for  instrumentation  to  permit  diagnosis  of  trouble. 

•  Allow  time  between  major  tests  to  analyze  for  failures  and  implement  corrective  action. 

13.2.8  TRMC/MRTFB 

Under  the  provision  of  section  196  of  Reference  (b),  the  DoD  established  the  Test  Resource 
Management  Center  (TRMC)  with  the  Director  reporting  directly  to  the  USD(AT&L)  without  the 
interposition  of  any  other  supervising  official.  The  Director,  TRMC  provides  oversight  for  T&E  strategic 
planning  and  budgets  for  the  Departments  T&E  activities,  including  the  MRTFB,  Central  Test  and 
Evaluation  Investment  Program  (CTEIP),  and  the  T&E  Science  and  Technology  Program. 

All  Services  operate  ranges  and  test  facilities.  The  MRTFB  is  a  national  asset  which  shall  be  sized, 
operated,  and  maintained  primarily  for  DoD  T&E  support  missions,  but  also  is  available  to  all  users 
having  a  valid  requirement  for  its  capabilities.  MRTFB  consists  of  a  broad  base  of  T&E  activities  managed 
and  operated  under  uniform  guidelines  to  provide  T&E  support  to  DoD  Components  responsible  for 
developing  or  operating  materiel  and  weapon  systems.  The  MRTFB  sites  are  displayed  in  Figure  13-1  by 
DoD  Component. 
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Figure  13-1  DoD  MRTB  by  DoD  Component 

MRTFB  facilities  are  available  for  use  by  all  the  Services,  U.S.  Government  agencies,  and  in  certain  cases, 
allied  foreign  governments  and  contractor  organizations.  Scheduling  is  based  on  a  priority  system,  and 
costs  for  usage  are  billed  uniformly.  The  Director,  TRMC  sets  policy  for  the  composition,  use,  and  test 
program  assignments  of  the  MRTFB.  The  individual  Services  fund,  manage,  and  operate  their  activities; 
they  are  reimbursed  for  direct  costs  of  testing  by  each  user  of  the  activity.  TRMC  sponsors  a  Joint  Test 
Assets  Database  that  lists  MRTFB  and  OTA  test  facilities,  test  area  and  range  data,  instrumentation,  and 
test  systems. 

Currently,  24  sites  comprise  the  MRTFB  across  the  United  States  and  its  territories,  including  many 
Government  T&E  facilities.  The  goal  is  for  PMs  to  use  Government  T&E  facilities  rather  than  building  or 
making  capital  investments  in  T&E  capabilities.  An  important  consideration  for  the  PM  focuses  on  the 
costs  associated  with  the  use  of  MRTFB  and  other  Government  facilities  compared  with  contractor 
facilities.  PMs  should  remain  aware  of  and  consider  the  use  of  MRTFBs  to  save  valuable  resources. 

The  Director,  TRMC  reviews  the  proposed  T&E  budgets  of  each  Military  Department  and  Defense 
Agency  with  T&E  responsibilities,  in  accordance  with  section  196(e)(2)  of  Reference  (b).  Following  the 
review,  and  not  later  than  January  31  of  the  year  preceding  the  fiscal  year  for  which  such  budgets  are 
proposed,  the  Director  must  submit  to  the  SecDef  a  report  containing  the  Directors  analysis  of  all  the 
proposed  T&E  budgets  and  determination  as  to  whether  these  proposed  budgets  are  adequate  and 
provide  balanced  support  for  the  Strategic  Plan  for  DoD  T&E  Resources. 
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TRMC  generates  an  annual  T&E  budget  certification  report  focused  on  the  MRTFB.  TRMC  coordinates 
budget  issues  that  lead  up  to  determining  the  Departments  budget  and  serves  as  the  SME  on  MRTFB 
funding  and  the  MRTFB  portions  of  Reference  (aq) . 

13.2.9  CTEIP 

TRMC  manages  the  implementation  of  and  executes  the  CTEIP.  The  CTEIP  provides  OSD  funding  and  a 
mechanism  for  the  development  and  acquisition  of  new  test  capabilities  to  satisfy  multi-Service  testing 
requirements.  It  was  established  in  1990  to  improve  the  coordination  and  planning  of  DoD  T&E 
capability  investments.  The  program  invests  in  T&E  capabilities  that  will  meet  the  test  requirements  of 
more  than  one  Service.  CTEIP  funds  about  50  projects  or  subprojects  at  any  given  time,  all  of  which  are 
in  various  states  of  development.  These  projects  range  from  quick  assessments  of  new  technologies  to 
full-scale  efforts  to  develop  new  test  capabilities.  Though  CTEIP  operates  under  the  oversight  of  TRMC, 
the  Services  and  Defense  Agencies  propose  and  execute  CTEIP  projects.  CTEIP  provides  a  coordinated 
process  for  funding  T&E  investments  that  leverage  Service  investments,  ensure  joint  development,  and 
follow  best  practices.  CTEIP  processes  also  encourage  reuse  and  the  promulgation  and  use  of  standards. 

13.2.10  Service  Test  Facilities 

There  are  other  test  resources  available  besides  the  MRTFB.  National  Guard  or  Reserve  units, 
commercial  or  international  test  facilities,  training  ranges  that  are  not  part  of  the  MRTFB,  non-DoD 
governmental  agencies  (e.g..  National  Aeronautics  and  Space  Administration  (NASA)  ranges),  and  war 
reserve  assets  may  be  available  to  support  DoD  T&E.  Testers  can  determine  resources  available  by 
contacting  their  Service  headquarters  staff  element.  Within  the  Army,  consult  documents  such  as  the 
ATEC  database  on  existing  Army  major  test  facilities,  major  instrumentation,  and  test  equipment;  the 
Project  Manager  for  Instrumentation,  Targets,  and  Threat  Simulators  (PM,  ITTS)  Targets  Information 
Manual  for  a  descriptive  catalogue  of  Army  targets  and  foreign  ground  assets  available  (or  in 
development)  for  support  of  T&E  and  training;  and  the  PM  ITTS  Threat  Inventory  Database  on  available 
assets  for  both  hardware  simulators  and  software  simulation  systems.  Information  on  specific  Navy  test 
resources  is  found  in  user  manuals  published  by  each  range. 

13.3  Test  Resource  Planning 

The  development  of  special  test  resources  to  support  a  weapon  system  test  can  be  costly  and  time- 
consuming.  This  development,  coupled  with  the  competition  for  existing  test  resources  and  facilities, 
requires  that  early  planning  be  accomplished  to  determine  all  test  resource  requirements  for  weapon 
system  T&E.  The  tester  must  use  Government  facilities  whenever  possible.  One  problem  associated  with 
range  and  facility  planning  is  that  major  defense  systems  tend  to  get  top  priority.  Range  schedules  are 
often  in  conflict  because  of  unforeseen  system  problems,  which  cause  schedule  delays  during  testing, 
and  if  funds  for  possible  retesting  have  not  been  adequately  programmed,  there  is  often  a  shortage  of 
funds  to  complete  testing. 
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13.3.1  TEMP  Resource  Requirements 

All  key  test  resource  requirements  should  be  stated  in  the  TEMP  and  include  items  such  as  unique 
instrumentation,  threat  simulators,  surrogates,  targets,  and  test  articles.  Part  IV  of  the  TEMP  discusses 
test  resources,  as  depicted  in  Figure  13-2.  Because  the  first  TEMP  must  be  prepared  for  program 
initiation,  initial  test  resource  planning  must  be  accomplished  very  early  as  part  of  the  TEMP  preparation 
process.  Refinements  and  reassessments  of  test  resource  requirements  are  included  in  each  TEMP 
update.  The  guidance  for  the  content  of  the  test  resource  summary  (Part  IV)  of  the  TEMP  is  outlined  in 
Reference  (j) .  Once  the  test  resource  requirements  are  identified,  the  PM  must  then  work  within  the 
Service  headquarters  and  range  management  structure  to  ensure  that  the  assets  are  available  when 
needed. 


Part  IV  -  Resource  Summary 

4.1  Introduction.  In  this  section,  specify  the  resources  necessary  to  accomplish  the  T&E  program.  Testing  will  be  planned  and 
conducted  to  take  full  advantage  of  existing  DoD  investment  in  ranges,  facilities,  and  other  resources  wherever  practical.  Provide  a 
list  in  a  table  format  including  schedule  of  all  key  test  and  evalustion  resources,  both  government  and  contractor,  that  will  be  used 
during  the  course  of  the  current  increment.  Include  long-lead  items  for  the  next  incrememnt  if  known.  Specifically,  identify  the 
following  test  resources  and  identify  any  shortfalls,  impact  on  planned  testing,  and  plan  to  resolve  shortfalls. 

4.1.1.  Test  Articles. 

4.1.2.  Test  Sites  and  Instrumentation. 

4.1.3.  Test  Support  Equipment. 

4.1.4.  Threat  Representation. 

4.1.5.  Test  Targets  and  Expendables. 

4.1.6.  Operational  Force  Test  Support. 

4.1.7.  Models,  Simulations,  and  Testbeds. 

4.1.8.  Joint  Mission  Environment. 

4.1.9.  Special  Requirements. 

4.2.  Federal,  State,  and  local  Requirements. 

4.3.  Manpower/Personnel  and  Training. 

4.4.  Test  Funding  Summary. 

Defense  Acquisition  Guidebook,  ANNEX:  TEST  AND  EVALUATION  MASTER  PLAN 


Figure  13-2  TEMP  PART  IV  -  Resource  Summary 
13.3.2  Service  Test  Resource  Planning 

More  detailed  listings  of  required  test  resources  are  generated  in  conjunction  with  the  detailed  test 
plans.  These  test  plans  describe  test  objectives,  MOEs,  test  scenarios,  and  specific  test  resource 
requirements. 
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13.3.2.1  Army  Test  Resource  Planning 

In  the  Army,  the  independent  evaluator,  with  the  assistance  of  the  developmental  and  operational 
testers,  prepares  input  to  the  TEMP  and  develops  a  SEP,  which  is  the  primary  planning  documents  for 
integrated  T&E  of  the  system.  These  documents  should  be  prepared  early  in  the  acquisition  cycle  (at  the 
beginning  of  system  acquisition  activities).  They  describe  the  entire  T&E  strategy  including  critical  issues, 
test  methodology,  MOEs,  and  all  significant  test  resources.  The  TEMP  and  SEP  provide  the  primary  input 
to  the  OTP,  which  contains  a  detailed  description  of  each  identified  required  test  resource,  where  and 
when  the  test  resource  is  to  be  provided,  and  the  providing  organization. 

The  tester  must  coordinate  the  OTP  with  all  major  commands  or  agencies  expected  to  provide  test 
resources.  Then  the  OTP  is  submitted  to  ATEC  for  review  by  the  TSARC  and  for  incorporation  into  the 
Army's  Five-Year  Test  Program  (FYTP).  The  initial  OTP  for  each  test  should  be  submitted  to  the  TSARC  as 
soon  as  testing  is  identified  in  the  TEMP.  Revised  OTPs  are  submitted  as  more  information  becomes 
available  or  requirements  change,  but  a  final  comprehensive  version  of  the  OTP  should  be  submitted  at 
least  18  months  before  the  resources  are  required. 

The  TSARC  is  responsible  for  providing  high-level,  centralized  management  of  T&E  resource  planning. 

The  TSARC  is  chaired  by  the  Commanding  General,  ATEC,  and  consists  of  a  general  officer  or  equivalent 
representatives  from  the  Army  staff  and  major  commands.  The  TSARC  meets  semiannually  to  review  all 
OTPs,  resolve  conflicts,  and  coordinate  all  identified  test  resource  requirements  for  inclusion  in  the 
FYTP.  The  FYTP  is  a  formal  resource  tasking  document  for  current  and  near-term  tests  and  a  planning 
document  for  tests  scheduled  for  the  out-years.  All  OTPs  are  reviewed  during  the  semiannual  reviews  to 
ensure  that  any  refinements  or  revisions  are  approved  by  the  TSARC  and  reflected  in  the  FYTP. 

The  TSARC-approved  OTP  is  a  tasking  document  by  which  the  tester  requests  Army  test  resources.  The 
TSARC  coordinates  resource  requests,  sets  priorities,  resolves  conflicts,  and  schedules  resources.  The 
resultant  FYTP,  when  approved  by  the  FIQDA  Deputy  Chief  of  Staff,  G3,  is  a  formal  tasking  document 
that  reflects  the  agreements  made  by  the  resource  providers  (AMC,  TRADOC,  Forces  Command,  etc.)  to 
make  the  required  test  resources  available  to  the  designated  tests.  If  test  resources  from  another 
Service,  a  non-DoD  governmental  agency  (such  as  the  Department  of  Energy  or  NASA),  or  a  contractor 
are  required,  the  request  is  coordinated  by  ATEC.  For  example,  the  request  for  a  range  must  be  made  at 
least  2  years  in  advance  to  ensure  availability.  Flowever,  due  to  the  long  lead  time  required  to  schedule 
these  non-Army  resources,  their  availability  cannot  be  guaranteed  if  testing  is  delayed  or  retesting  is 
required.  The  use  of  resources  outside  the  United  States  such  as  in  Canada,  Germany,  or  other  NATO 
countries,  is  also  handled  by  ATEC. 

13.3.2.2  Navy  Test  Resource  Planning 

In  the  Navy,  the  developing  agency  and  the  operational  tester  are  responsible  for  identifying  the  specific 
test  resources  required  in  testing  the  weapon  system.  In  developing  requirements  for  test  resources,  the 
PM  and  OTD  refer  to  capability  documents,  the  programs  Acquisition  Strategy,  threat  assessments,  and 
Secretary  of  the  Navy  Instruction  (SECNAVINST)  5000  series,  among  other  documents.  Upon  approval, 
the  TEMP  becomes  the  controlling  management  document  for  all  T&E  of  the  system.  It  constitutes 
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direction  to  conduct  the  T&E  program  defined  in  the  TEMP,  including  the  commitment  of  RDT&E 
financial  support  and  of  fleet  units  and  schedules.  The  TEMP  is  prepared  by  the  PM,  who  is  provided 
OT&E  input  by  the  COMOPTEVFOR  OTD.  The  TEMP  defines  all  T&E  (DT&E,  OT&E,  and  PAT&E)  to  be 
conducted  for  the  system  and  describes,  in  as  much  detail  as  possible,  the  test  resources  required. 

The  Navy  uses  its  operational  naval  forces  to  provide  realistic  T&E  of  new  and  upgraded  weapon 
systems.  Each  quarter,  CNO  (N84)  and  the  COMOPTEVFOR  fleet  scheduling  officer  solicit  RDT&E  fleet 
support  requirements  from  all  Navy  acquisition  programs.  RDT&E  fleet  support  requests  are  to  be 
submitted  at  least  9  months  in  advance  of  required  support.  COMOPTEVFOR  uses  the  Unclassified  Test 
and  Evaluation  System  to  organize  RDT&E  fleet  requests.  RDT&E  fleet  requests  inside  of  9  months  are 
considered  emergent  and  must  be  requested  through  record  message  traffic.  The  COMOPTEVFOR  fleet 
scheduling  officer  compiles  all  requests  and  works  with  the  fleet  to  schedule  requested  assets.  CNO 
(N84)  may  be  called  upon  to  prioritize  RDT&E  fleet  asset  requests.  For  additional  information  regarding 
Navy  fleet  RDT&E  support,  see  SECNAV  M-5000.2  (Reference  (ap)).  Requests  for  use  of  range  assets  are 
usually  initiated  informally  with  a  phone  call  from  the  PM  and/or  OTD  to  the  range  manager  and 
followed  by  formal  documentation.  Use  of  most  Navy  ranges  should  be  scheduled  at  least  a  year  in 
advance.  Each  range  consolidates  and  prioritizes  user  requests,  negotiates  conflicts,  and  attempts  to 
schedule  range  services  to  satisfy  all  requests. 

The  COMOPTEVFOR  is  authorized  direct  liaison  with  other  Service-independent  OTAs  to  obtain  non- 
Navy  OTA-controlled  resources.  Requests  for  other  Government-owned  resources  are  forwarded  to  the 
CNO  (N84)  for  formal  submission  to  the  Service  Chief  (for  Service  assets)  or  to  the  appropriate 
Government  agency  (e.g..  Department  of  Energy  or  NASA).  Use  of  contractor  resources  is  usually 
handled  by  the  PM,  although  contractor  assets  are  seldom  required  in  OT&E  because  the  fleet  is  used  to 
provide  an  operational  environment.  Requests  for  use  of  foreign  ranges  are  handled  by  the  PM  through 
the  Navy  International  Programs  Office. 

13.3.2.3  Air  Force  Test  Resource  Planning 

The  test  resources  required  for  OT&E  of  an  Air  Force  weapon  system  are  identified  in  detail  in  the  TRP, 
which  is  prepared  by  the  responsible  Air  Force  OT&E  organization.  In  general,  AFOTEC  is  the  test 
organization  for  OT&E  programs,  but  other  MAJCOMs  also  execute  OT.  AFOTEC  obtains  support  from  a 
Service  MAJCOM  test  agency  for  non-major  programs,  with  AFOTEC  directing  and  providing  assistance, 
as  required. 

During  the  advanced  planning  phase  of  a  weapon  system  acquisition  (5  to  6  years  before  OT&E),  AFOTEC 
prepares  the  OT&E  section  of  the  first  full  TRP,  coordinates  the  TRP  with  all  supporting  organizations, 
and  assists  the  resource  manager  (RM)  in  programming  required  resources.  The  resource  requirements 
listed  in  the  resource  information  network  TRP  are  developed  by  the  test  manager,  RM,  and  test  support 
group,  using  sources  such  as  the  Operational  Requirements  Document  and  threat  assessments.  The  TRP 
should  specify,  in  detail,  all  the  resources  necessary  to  successfully  conduct  a  test  once  it  is  entered  in 
the  Test  Resource  Information  Management  System  (TRIMS). 
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The  TRP  is  the  formal  means  by  which  test  resource  requirements  are  communicated  to  the  Air  Staff  and 
to  the  appropriate  commands  and  agencies  tasked  to  supply  the  needed  resources.  Therefore,  if  a 
required  resource  is  not  specified  in  the  TRP,  it  is  likely  the  resource  will  not  be  available  for  the  test. 

The  TRP  is  revised  and  updated  on  a  continuous  basis  because  the  test  resource  requirements  become 
better  defined  as  the  OT&E  plans  mature.  The  initial  TRP  serves  as  a  baseline  for  comparison  of  planned 
OT&E  resources  with  actual  expenditures.  Comparisons  of  the  initial  TRP  with  subsequent  updates 
provide  an  audit  trail  of  changes  in  the  test  program  and  its  testing  requirements.  AFOTEC  maintains  all 
TRPs  on  TRIMS  to  permit  immediate  response  to  all  queries  regarding  test  resource  requirements. 

The  AFOTEC  RM  consolidates  the  resource  requirements  from  all  TRPs,  coordinating  with  participating 
and  supporting  organizations  and  agencies  outside  AFOTEC.  Twice  yearly,  the  RM  office  prepares  a  draft 
of  the  U.S.  Air  Force  (USAF)  Program  for  Operational  Test  document.  It  is  a  master  planning  and 
programming  document  for  resource  requirements  for  all  Fleadquarters  (FIQ)  USAF-directed  OT&E  and 
is  distributed  to  all  concerned  commands,  agencies,  and  organizations  for  review  and  coordination.  It  is 
then  submitted  to  the  Air  Staff  for  review. 

All  requests  for  test  resources  are  coordinated  by  FIQ  AFOTEC  as  part  of  the  TRP  preparation  process. 
When  a  new  weapon  system  development  is  first  identified,  AFOTEC  provides  a  test  manager  (TM)  who 
begins  long-term  OT&E  planning.  The  TM  begins  identifying  needed  test  resources,  such  as 
instrumentation,  simulators,  and  models,  and  works  with  the  resources  directorate  to  obtain  them.  If 
the  required  resource  does  not  belong  to  AFOTEC,  then  AFOTEC  will  negotiate  with  the  commands 
having  the  resource.  In  the  case  of  models  and  simulators,  AFOTEC  surveys  what  resources  are  available, 
assesses  credibility,  and  then  coordinates  with  the  owner  or  developer  to  use  them.  The  Joint  Technical 
Coordinating  Group  (JTCG)  publishes  a  document  on  electronic  warfare  (EW)  models. 

Range  scheduling  should  be  done  early.  At  least  a  year  is  required,  but  often  a  test  can  be 
accommodated  with  a  few  months  notice  if  there  is  no  requirement  for  special  equipment  or 
modifications  to  be  provided  at  the  range.  Some  of  the  Air  Force  ranges  are  scheduled  well  in  advance 
and  cannot  accommodate  tests  that  encounter  delays  or  retest  requirements. 

The  RM  attempts  to  resolve  conflicts  among  various  systems  competing  for  scarce  test  resources  and 
elevates  the  request  to  the  Commander,  AFOTEC,  if  necessary.  Decisions  on  resource  utilization  and 
scheduling  are  based  on  the  weapon  systems  assigned  priority. 

The  RM  and  the  TM  also  arrange  for  use  of  the  resources  of  other  Services,  non-DoD  Government 
agencies,  and  contractors.  Use  of  non-U. S.  resources,  such  as  a  Canadian  range,  are  coordinated  by 
AF/TE  and  based  on  formal  MOUs.  The  U.S.  Air  Forces  in  Europe/Directorate  of  Operations-Operations 
handles  requests  for  European  ranges.  Use  of  a  contractor-owned  resource,  such  as  a  model,  is  often 
obtained  through  the  system  program  office  (SPO)  or  a  general  support  contract. 

The  Air  Force  conducts  FDEs,  which  are  a  type  of  OT&E  performed  by  MAJCOM  operational  test 
organizations  in  support  of  MAJCOM-managed  system  acquisition-related  decisions  prior  to  initial 
fielding,  or  for  MAJCOM  sustainment  or  upgrade  activities.  An  FDE  may  be  used  for  multiple  purposes; 
for  example,  an  FDE  may  be  used  to: 
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•  Evaluate  and  verify  the  resolution  of  previously  identified  deficiencies  or  shortfalls,  including 
those  rated  in  AFOTECs  OT&E  final  report  as  not  having  a  substantial  or  severe  impact  on 
mission  operations. 

•  Evaluate  routine  software  modifications  (e.g.,  operational  flight  programs),  subsequent 
increments,  upgrades,  and  other  improvements  or  changes  made  to  sustain  or  enhance  the 
system. 

•  Evaluate  and  verify  correction  of  new  performance  shortfalls  discovered  after  fielding  of  the 
system. 

•  Evaluate  operational  systems  against  foreign  equipment. 

•  Evaluate  operational  systems  against  new  or  modified  threats. 

•  Evaluate  military-unique  portions  and  applications  of  COTS,  NDI,  and  Government-furnished 
equipment  (GFE)  for  military  use. 

13.4  Test  Resource  Funding 

The  FYDP,  incorporating  an  annual  budgeting  process,  is  the  basic  DoD  programming  document  that 
records,  summarizes,  and  displays  SecDef  decisions.  In  the  FYDP,  costs  are  divided  into  three  categories 
for  each  acquisition  PE:  R&D  costs,  investment  costs,  and  operating  costs.  Congress  appropriates  to  the 
Office  of  Management  and  Budget  (0MB),  and  0MB  apportions  funding  through  the  SecDef  to  the 
Services  and  to  other  Defense  Agencies.  The  Services  and  Defense  Agencies  then  allocate  funds  to 
others  (claimants,  sub-claimants,  administering  offices,  commanding  generals,  etc.). 

The  PPBE  system  is  a  DoD  internal  system  used  to  develop  input  for  Congress  for  each  year's  budget 
while  developing  future-year  budgets.  PPBE  is  calendar  oriented.  There  are  concurrent  PPBE  cycles 
ongoing  at  one  time.  These  cycles  are  planning,  programming,  budgeting,  and  execution.  At  any  one 
time,  there  are  multiple  budgets  being  worked  by  the  Services.  The  current  1-year  budget  is  being 
executed.  The  next  several  years  of  defense  planning  are  being  programmed,  and  long-range  program 
plans  and  planning  guidance  are  being  reviewed  for  updating.  Figure  13-3  shows  resourcing  cycle 
overlaps  in  the  DoD  budget  process. 
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Resource  Allocation  Process  Overlap 
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Figure  13-3  Resourcing  Cycle  Overlaps  in  the  DoD  Budget  Process 
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There  are  various  types  of  funding  in  the  PPBE  process:  R&D  funding  for  maintaining  the  technology 
base;  exploratory  development  funding  for  conducting  the  concept  assessments;  advanced 
development  funding  for  conducting  both  the  concept  development  and  the  early  prototyping; 
engineering  development  funding  for  demonstrating  the  EDM;  and  procurement  funding  for  conducting 
LRIP,  FRP,  system  deployment,  and  operational  support.  RDT&E  management  and  support  funding  is 
used  throughout  the  development  cycle  until  the  system  is  operationally  deployed  when  O&M  funding 
is  used.  The  RDT&E  appropriation  funds  the  costs  associated  with  R&D  intended  to  improve 
performance,  including  test  items,  DT&E  and  test  support  of  OT&E  of  the  system,  or  equipment  test 
items. 


Funding  that  is  planned,  programmed,  and  budgeted  through  the  PPBE  cycle  is  not  always  the  same 
funding  amount  that  Congress  appropriates  or  the  PM  receives.  If  the  required  funding  for  a  test 
program  is  not  appropriated  by  Congress,  the  PM  has  limited  courses  of  action.  The  PM  can  try  to 
reprogram  the  needed  funds  or  submit  the  shortfall  as  a  requirement  on  an  omnibus  supplemental  bill. 

Generally,  testing  that  is  accomplished  for  a  specific  system  before  the  production  decision  is  funded 
from  RDT&E  appropriations,  and  testing  that  is  accomplished  after  the  production  decision  is  funded 
from  other  RDT&E,  procurement,  or  O&M  appropriations.  Testing  of  Pis,  block  upgrades,  and  major 
modifications  is  funded  from  the  same  appropriations  as  the  program  development.  FOT&E  is  normally 
funded  from  RDT&E  or  O&M  funds. 
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Funding  associated  with  T&E  (including  instrumentation,  targets,  and  simulations)  is  identified  in  the 
system  acquisition  cost  estimates.  Service  acquisition  plans,  and  the  TEMP.  General  funding  information 
for  developmental  and  operational  tests  is  provided  in  the  following  subsections. 

13.4.1  DT  Funding 

Funds  required  for  the  conduct  of  engineering  and  DTs  developmental  tests  are  programmed  and 
budgeted  by  the  materiel  developer  as  early  as  the  MSA  phase  of  acquisition  and  updated  based  upon 
the  requirements  of  the  TEMP.  These  costs  may  include  but  are  not  limited  to  procuring  test 
samples/prototypes;  support  equipment;  transportation  costs;  technical  data;  training  of  test  personnel; 
repair  parts;  and  test-specific  instrumentation,  equipment,  and  facilities.  The  DT&E  funds  are  expended 
for  contractor  and  Government  DT  activities. 

The  Service  PM  may  be  required  to  pay  for  the  use  of  test  resources,  such  as  the  MRTFB,  and  for  the 
development  of  specialized  resources  needed  specifically  for  testing  the  weapon  system  being 
developed. 

13.4.2  OT  Funding 

Funds  required  to  conduct  OT  are  usually  programmed  and  budgeted  by  the  Service  PM  organization. 
The  funds  are  programmed  in  the  Services  long-range  test  program,  and  the  funds  requirements  are 
obtained  from  the  test  resourcing  documentation  and  TEMP.  The  Air  Force  funds  OT&E  separate  from 
the  program  office  through  a  dedicated  PE  for  AFOTEC-conducted  OT  and  through  MAJCOM  funding  for 
MAJCOM-funded  testing. 

13.4.3  Army  Funding 

Test  resources  are  developed  and  funded  under  various  Army  appropriations.  Direct-cost  funding 
requires  that  test-specific  (direct)  costs  associated  with  a  particular  test  program  be  reimbursed  by  the 
program  office  to  the  designated  test  agency.  The  AMC  and  its  commodity  commands  provide  test 
items,  spare  parts,  support  items  (such  as  diagnostic  equipment),  instrumentation,  and  expendables. 
Soldiers,  ranges,  fuel,  test  support  personnel,  and  maneuver  areas  are  provided  by  U.S.  Army  TRADOC  or 
U.S.  Army  Forces  Command  (FORSCOM).  Weapon  system  PMs  use  RDT&E  funds  to  reimburse  these 
supporting  commands  for  costs  directly  related  to  their  tests.  Weapon  system  materiel  developers  are 
also  responsible  for  funding  the  development  of  new  test  resources  specifically  needed  to  test  the 
weapon  system.  Examples  of  such  special-purpose  resources  include  models,  simulations,  special 
instrumentation  and  test  equipment,  range  modifications,  EW  simulators,  and  sometimes  threat 
simulators.  Although  the  Army  has  a  separate  budget  and  development  plan  for  threat  simulators-the 
PM  ITTS  threat  simulators  program-many  weapon  system  developers  still  have  to  fund  the  cost  of  new 
threat  systems  that  are  specifically  needed  to  test  their  weapon  system.  Funding  for  Army  OT  is  through 
the  PMs  PE  and  is  given  to  ATEC  for  direct  control  of  funds  for  each  program.  Funding  requirements  are 
developed  in  consonance  with  the  OTP. 
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13.4.4  Navy  Funding 

In  the  Navy,  the  weapon  system  PM  is  responsible  for  funding  the  development  of  all  required  test- 
specific  resources  from  the  programs  RDT&E  funds.  Direct-cost  funding  requires  that  test-specific 
(direct)  costs  associated  with  a  particular  test  program  be  reimbursed  by  the  program  office  to  the 
designated  test  agency.  These  resources  include  test  articles,  expendables,  one-of-a-kind  targets,  data 
collection/reduction,  and  instrumentation.  The  development  of  generic  test  resources  that  can  be  used 
in  OT&E  of  multiple  weapon  systems,  such  as  targets,  threat  simulators,  and  range  capabilities,  is  funded 
from  the  Office  of  the  Chief  of  Naval  Operations  generic  accounts  (such  as  target  development)  and  not 
from  weapon  systems  RDT&E.  The  PMs  RDT&E  funds  pay  for  all  DT  and  OT  through  lOT&E.  The  PM  pays 
for  all  post-production  OT  with  program  funds. 

13.4.5  Air  Force  Funding 

In  the  Air  Force,  direct-cost  funding  requires  that  test-peculiar  (direct)  costs  associated  with  a  particular 
test  program  be  reimbursed  by  the  SPO  to  the  designated  test  agency.  The  RDT&E  appropriation  funds 
the  cost  associated  with  R&D,  including  test  items,  spare  parts,  expendables,  instrumentation  and 
ranges,  DT&E,  and  AFMC  support  of  OT&E  of  the  system  or  equipment  and  the  test  items.  Costs 
associated  with  lOT&E  are  RDT&E-funded,  and  costs  of  QOT&E  are  O&M-funded.  AFOTEC  is  funded 
through  its  own  PE  and  has  direct  control  of  OT&E  funds  for  all  programs.  The  lOT&E  manager  prepares 
a  TRP  that  summarizes  the  resource  requirements  for  lOT&E  and  related  test  support.  All  pre-test  lOT&E 
planning  is  budgeted  through  and  paid  out  of  the  O&M  appropriation.  The  FOT&E  costs  are  paid  by 
AFOTEC  and/or  the  MAJCOM  operating  the  system  and  funded  by  the  O&M  appropriation. 

13.4.6  TRMC  Certification  of  Certain  Funding 

According  to  Section  196  of  Reference  (b),  the  SecDef,  acting  through  the  Under  Secretary  of  Defense 
(Comptroller)/Chief  Financial  Officer  (USD(C)/CFO),  requires  that  the  Director,  TRMC  report  annually  the 
certification  of  the  DoD  test  budgets.  The  Secretaries  of  the  Military  Departments  and  the  Directors  of 
Defense  Agencies  with  T&E  responsibilities  must  transmit  such  Secretary's  or  Director's  proposed 
budget  for  T&E  activities  for  a  fiscal  year  to  the  Director,  TRMC  for  review  before  submitting  such 
proposed  budget  to  the  USD(C)/CFO. 

13.5  Summary 

Test  resources  have  many  conflicting  demands,  and  their  use  must  be  scheduled  well  in  advance  of  a 
test.  Resources  specific  to  a  particular  test  must  often  be  developed  and  funded  from  the  PMs  own 
RDT&E  budget.  PMs  must  use  DoD  Government  T&E  capabilities  and  invest  in  Government  T&E 
infrastructure  unless  an  exception  can  be  justified  as  cost-effective  to  the  Government.  Thus,  the  PM 
and  testers  must  ensure  that  test  resource  requirements  are  identified  early  in  the  acquisition  cycle, 
that  they  are  documented  in  the  initial  TEMP,  and  that  modifications  and  refinements  are  reported  in 
the  TEMP  updates. 
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Funds  for  testing  are  provided  by  congressional  appropriation  to  the  0MB,  which  apportions  the  funds 
to  the  Services  through  the  SecDef.  PPBE  is  the  DoD  process  used  to  formulate  budget  requests  to 
Congress.  Requests  by  PMs  for  test  resources  are  usually  outlined  in  the  TEMP.  PMs  must  conduct  a  CBA 
when  using  and  investing  in  non-Government  T&E  capabilities  and  infrastructure  and  must  document 
the  CBA  assumptions  and  results  in  an  approved  TEMP  before  proceeding  with  the  program.  Programs 
on  DT&E  oversight  must  provide  the  DASD(DT&E)  with  the  results  of  the  CBA  for  approval.  Generally, 
system  development  is  funded  from  RDT&E  funds  until  the  system  is  operationally  deployed  and 
maintained.  O&M  funds  are  used  for  FOT&E  and  system  maintenance.  The  weapon  system  materiel 
developer  is  also  responsible  for  funding  the  development  of  new  test  resources  specifically  needed  to 
test  the  weapon  system. 
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Test  &  Evaluation  Management  Guide 

Chapter  14  -  Introduction  to  Acquisition  of  Defense  Business  Systems  (DBSs) 

14.1  Introduction 

This  chapter  describes  the  incrementai  acquisition  approach  for  DBSs.  it  introduces  the  Business 
Capabiiity  Lifecycie  (BCL)  Modei  and  discusses  in  more  detaiis  the  BCL  Acquisition  Business  Modei.  it 
describes  how  the  BCL  Acquisition  Business  Modei  supports  the  impiementation  of  the  BCL  and  depicts 
the  phases,  miiestones,  and  decision  points  of  the  BCL  acquisition  process. 

14.2  DBSs 

As  stated  in  Section  2222(i)(2)  of  Reference  (b),  the  term  defense  business  system  means  an  information 
system,  other  than  a  nationai  security  system,  operated  by,  for,  or  on  behaif  of  the  DoD,  inciuding 
financiai  systems,  mixed  systems,  financiai  data  feeder  systems,  and  iT  and  iA  infrastructure,  used  to 
support  business  activities,  such  as  acquisition,  financiai  management,  iogistics,  strategic  pianning  and 
budgeting,  instaiiations  and  environment,  and  human  resource  management. 

iT  and  iA  infrastructures  that  are  intended  to  be  used  to  support  the  generai  activities  of  the 
Department  (i.e.,  not  soieiy  dedicated  to  a  specific  business  system)  are  not  DBSs.  These  iT  and  iA 
infrastructures  inciude  iocai  area  networks,  metropoiitan  area  networks,  wide  area  networks,  and 
teiephone  systems. 

14.3  BCL  Model 

The  BCL  model  in  Reference  (n)  is  the  directed  acquisition  process  for  DBSs.  DBSs  are  validated  by  the 
Defense  Business  Systems  Management  Committee  (DBSMC).  These  systems  employ  a  business  case 
document  using  the  BCL  process  in  lieu  of  JCIDS  documents  for  the  capability  requirements  and 
associated  solutions.  The  principles  of  BCL  can  be  applied  at  the  increment  or  the  release  level-BCL 
provides  the  framework  for  structuring  the  definition,  development,  testing,  production,  deployment, 
and  support  of  DBSs.  This  model  is  a  guideline,  and  tailoring,  consistent  with  statute  and  sound  business 
practice,  is  encouraged.  As  stated  in  Reference  (n) .  it  is  DoD  policy  that : 

•  BCL  is  the  overarching  framework  for  the  planning,  design,  acquisition,  deployment,  operations, 
maintenance,  and  modernization  of  DBSs,  in  accordance  with  Section  2222(f)  of  Reference  (b). 
BCL  facilitates  DBS  acquisition  by  providing  a  process  tailored  to  the  unique  requirements  of 
business  systems. 

•  BCL  shall  apply  to  each  DBS  modernization  with  a  total  cost  over  $1,000,000. 

•  When  a  MAIS  DBS  employs  an  incremental  acquisition  approach,  all  functional  capabilities 
associated  with  a  given  increment  shall  be  reflected  in  any  resultant  APB  (cost,  performance, 
and  schedule)  and  must  be  achievable  within  5  years  from  when  funds  were  first  obligated.  For 
all  DBSs  that  are  not  MAIS  or  otherwise  designated,  they  must  achieve  IOC  within  5  years  from 
MS  A.  Delivery  of  capability  within  an  increment  (e.g.,  releases,  sub-phases,  software  drops) 
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must  be  based  on  technologies  that  have  been  determined  to  be  mature  at  the  MS  B  decision 
review.  Functional  capabilities  that  are  not  supported  by  adequate  cost  estimates,  mature 
technologies,  etc.,  shall  be  deferred  to  subsequent  program  increment(s). 

For  all  DBSs  that  meet  the  MAIS  threshold  or  have  been  designated  as  special  interest  or  other  major 
information  technology  investment  program  shall  be  subject  to  OSD  oversight.  For  all  DBSs  that  do  not 
meet  the  MAIS  threshold  or  have  not  been  otherwise  designated  as  special  interest  or  other  major 
information  technology  investment  program,  the  Fleads  of  the  DoD  Components  shall  provide  oversight 
of  their  acquisition  processes  and  procedures,  which  shall  be  consistent  with  applicable  statutes, 
regulations,  and  Reference  (n). 

14.4  Incremental  Acquisition  Approach  For  DBSs 

As  stated  in  Reference  (n) ,  approved  business  need  that  requires  a  materiel  solution  must  be  divided 
into  discrete,  fully  funded,  and  manageable  increments  to  facilitate  development  and  implementation. 
Each  increment  shall  be  a  useful  and  supportable  operational  capability  that  can  be  developed,  tested, 
produced,  deployed,  and  sustained.  The  principles  of  BCL  can  apply  at  the  increment  level  and  at  the 
release  level.  Thus,  there  may  be  multiple  releases  within  an  increment.  Multiple  increments  may  also 
be  approved  concurrently  if  they  have  well-defined  and  approved  requirements,  are  fully  funded,  and 
have  appropriate  entrance  and  exit  criteria,  and  if  the  MDAs  authorization  to  proceed  is  documented  in 
an  ADM.  To  facilitate  rapid  and  responsive  development,  no  more  than  12  months  shall  normally  elapse 
between  the  MDD  and  MS  A.  Following  MS  A,  no  more  than  12  months  shall  normally  elapse  between 
the  initial  contract  or  option  award  and  MS  B.  Following  MS  B,  no  more  than  18  months  shall  normally 
elapse  between  contract  or  option  award  and  the  full  deployment  decision  (FDD).  The  FDD  is  the  final 
decision  made  by  the  MDA  authorizing  an  increment  of  the  program  to  deploy  software  for  operational 
use  in  accordance  with  Section  2445(a)  of  Reference  (b).  Exceptions  must  be  reviewed  by  the 
responsible  Investment  Review  Board  (IRB)  and  approved  by  the  MDA.  The  MDA  shall  not  grant  a  MS  A 
decision  if  IOC  cannot  be  achieved  within  5  years  and  in  no  event  shall  FDD  occur  later  than  5  years  from 
when  funds  were  first  obligated  for  the  program  in  accordance  with  Section  811  of  Public  Law  109-364 
(Reference  (aq)). 

14.4.1  BCL  Acquisition  Business  Model 

The  BCL  acquisition  business  model  shown  in  Figure  14-1  supports  the  implementation  of  BCL  and 
depicts  the  phases,  milestones,  and  decision  points  of  the  BCL  acquisition  process. 

14.4.2  Business  Capability  Definition  (BCD)  Phase 

The  purpose  of  the  BCD  phase  is  to  analyze  a  perceived  business  problem,  capability  gap,  or  opportunity 
(hereinafter  referred  to  as  business  need)  and  document  the  results  in  a  problem  statement  to  inform 
IRB  Chair  and  MDA  decisions.  The  activities  performed  and  documentation  required  in  the  BCD  phase 
shall  be  used  in  lieu  of  the  JCIDS.  The  BCD  phase  begins  with  the  identification  of  a  business  need.  The 
business  need  can  be  identified  by  anyone  throughout  the  DoD  enterprise,  including  the  Combatant 
Commanders  (i.e.,  in  their  integrated  priority  lists)  and  capability  area  managers.  The  BCD  phase  ends 
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when  the  responsible  IRB  Chair  approves  the  problem  statement  and  the  approved  AoA  study  guidance 
and  AoA  study  plan  are  submitted  to  the  IRB  Chair. 
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Figure  14-1  BCL  Acquisition  Business  Model 
Investment  Management  (IM)  Phase 

The  purpose  of  the  IM  phase  is  to  assess  potential  materiel  solutions  and  to  satisfy  the  phase-specific 
entrance  criteria  designated  by  the  MDA  for  the  next  milestone.  The  IM  phase  begins  at  the  MDD;  the 
MDD  shall  be  mandatory  for  all  DBSs.  A  PM  is  assigned  for  each  acquisition  program  early  in  the  IM 
phase.  IM  phase  activities  include  the  analysis  necessary  to  describe  the  requirements  for  the  materiel 
solution;  the  solution  scope,  objectives,  business  outcomes,  outcome-based  performance  measures, 
constraints,  and  dependencies;  the  program  justification,  including  assumptions,  doctrine,  organization, 
training,  materiel,  leadership  and  education,  personnel,  and  facilities  (DOTMLPF)  impact,  critical  success 
factors,  risks,  detailed  cost  and  benefits  including  return  on  investment  analysis,  funding  profile,  and 
delivery  schedule;  and  an  acquisition  and  contracting  approach.  The  IM  phase  analysis  is  summarized  in 
a  business  case  developed  and  signed  by  the  functional  sponsor  and  PM.  The  business  case  includes  the 
problem  statement  and  the  results  of  the  IM  phase  analysis  and  serves  as  the  foundation  for  all  BCL 
efforts  and  decisions.  The  PM,  functional  sponsor,  and  T&E  community  jointly  develop  and  include  in  the 
business  case  a  plan  that  describes  but  is  not  limited  to  an  integrated  test  program  schedule;  test 
management  structure  and  processes;  DT&E  and  OT&E  phases  (objectives,  events,  entrance  criteria, 
scope,  and  limitations);  CTPs;  COIs,  with  associated  MOEs  and  MOPs;  and  required  resources.  The 
DOT&E  and  DASD(DT&E)  (or,  for  DBSs  that  do  not  meet  the  MAIS  threshold,  the  DoD  Component 
equivalents),  in  accordance  with  Public  Law  111-23  (Reference  (ar)),  approve  the  initial  test  plan  and 
updates  submitted  at  subsequent  decision  points.  For  MAIS  programs  and  MDAPs,  prior  to  the  MS  A 
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review,  the  DOT&E  and  DASD(DT&E)  jointly  approve  the  test  sections  of  the  business  case;  the  DASD(SE) 
approves  the  SE  sections  of  the  business  case.  For  MAIS  programs  and  MDAPs,  an  independent  risk 
assessment  known  as  Enterprise  Risk  Assessment  Methodology  (ERAM)  is  conducted  prior  to  MS  A  to 
review  the  results  of  phase  analysis.  As  a  result  of  the  ERAM,  the  PM  prepares  a  risk  mitigation  plan  for 
MDA  review  and  approval  at  MS  A.  The  IM  phase  ends  when  phase  requirements  are  satisfied,  the 
responsible  IRB  reviews  the  business  case,  and  the  responsible  IRB  Chair  forwards  an  MS  A 
recommendation  to  the  MDA. 

14.4.3  Prototyping  Phase 

The  purpose  of  the  prototyping  phase  is  to  demonstrate  the  capability  of  the  software  to  meet  business 
process  requirements  as  outlined  in  the  business  case.  Prototyping  includes  installing  IT  in  a  relevant 
environment  to  gain  the  knowledge  necessary  to  refine  user  requirements  and  inform  APB 
development.  The  prototyping  phase  begins  when  the  MDA  approves  the  business  case  and  documents 
the  MS  A  decision  in  an  ADM.  Following  MS  A,  no  more  than  12  months  shall  normally  elapse  between 
initial  contract  or  option  award  and  MS  B  unless  an  exception  has  been  approved  by  the  MDA  and 
documented  in  the  ADM.  For  MAIS  programs  and  MDAPs,  an  ERAM  is  conducted  prior  to  MS  B.  Based 
on  the  results  of  the  ERAM,  the  PM  prepares  a  risk  mitigation  plan  for  MDA  review  and  approval  at  MS 
B.  The  PM  compiles  a  MS  B  acquisition  decision  package  and  submits  it  to  the  responsible  IRB  (or,  for 
DBSs  that  do  not  meet  the  MAIS  threshold,  the  DoD  Component  equivalent  review  group)  for  review.  In 
addition  to  other  documents,  this  package  includes  an  updated  business  case  with  DOT&E  and 
DASD(DT&E)  joint  approval  of  the  test  sections  of  the  business  case  and  DASD(SE)  approval  of  the  SE 
sections  of  the  business  case  (for  MAIS  programs  and  MDAPs).  The  prototyping  phase  ends  when  phase 
requirements  are  satisfied  and  the  responsible  IRB  Chair  forwards  a  MS  B  recommendation  to  the  MDA. 

14.4.4  Engineering  Development  Phase 

The  purpose  of  the  engineering  development  phase  is  to  demonstrate  that  the  materiel  solution  for  the 
increment  has  been  designed,  configured,  developed,  and  tested  in  a  manner  consistent  with  the 
approved  business  case  and  program  charter  and  that  the  materiel  solution  is  ready  for  limited  fielding 
and  testing  in  an  operational  environment.  The  engineering  development  phase  begins  when  the  MDA 
approves  the  updated  business  case  and  the  APB  and  documents  the  decision  in  an  ADM.  During  the 
engineering  development  phase,  the  PM  refines  system  requirements,  configures  the  software,  builds 
functionality  as  required,  conducts  DT,  and  plans  for  OT.  The  PM  demonstrates  that  the  materiel 
solution  for  the  increment  has  been  designed,  configured,  developed,  tested,  and  evaluated  in  a  manner 
consistent  with  the  approved  business  case  and  program  charter  and  that  it  is  ready  to  be  proven  in  an 
operational  environment.  Following  MS  B,  no  more  than  18  months  shall  normally  elapse  between 
contract/option  award  and  FDD,  as  described  in  the  business  case  by  the  functional  sponsor,  unless  an 
exception  is  approved  by  the  MDA  and  documented  in  an  ADM.  The  test  community  tests  and  evaluates 
the  delivered  capability  to  determine  whether  it  adheres  to  the  outcomes  defined  in  the  business  case 
and  whether  it  is  compliant  with  the  Business  Enterprise  Architecture  (BEA).  For  MAIS  programs  and 
MDAPs,  DT  is  conducted  in  accordance  with  the  test  plan,  as  documented  in  the  business  case,  and 
approved  by  the  DASD(DT&E).  For  MAIS  programs  and  MDAPs,  OT  is  conducted  in  accordance  with  the 
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operational  test  plan  approved  by  the  DOT&E.  For  programs  on  OSD  T&E  oversight,  the  DoD 
Components  conduct  an  OTRR  before  commencing  OT  for  any  increment.  The  engineering  development 
phase  ends  when  phase  requirements  are  satisfied  and  when  the  functional  sponsor  reviews  the  test 
results  and  determines  that  the  outcomes  and  metrics  as  stated  in  the  approved  business  case  are 
satisfied. 

14.4.5  Limited  Fielding  Phase 

The  purpose  of  the  limited  fielding  phase  is  to  limit  risk  by  providing  the  capability  to  a  limited  number 
of  users  and  testing  it  in  an  operational  environment.  The  DOT&E  reports  on  the  adequacy  of  OT  and 
whether  that  testing  confirms  the  effectiveness  and  suitability  of  the  system.  Entrance  criteria  include 
completion  or  satisfaction  of  the  objectives  of  the  engineering  development  phase  (including  a 
developmentally  tested,  BEA-compliant,  production-representative  system,  ready  for  lOT&E);  the 
functional  sponsors  determination  that  the  capability  achieves  the  outcomes  specified  in  the  business 
case;  and  the  programs  compliance  with  the  statutory  and  regulatory  requirements  specified  for  MS  C. 
The  limited  fielding  phase  begins  when  the  functional  sponsor  and  the  MDA  approve  fielding  the 
capability  into  an  operational  environment  for  lOT&E  and  the  MDA  documents  the  decision  in  the  MS  C 
ADM.  The  PM  engages  an  OTA  to  verify  that  the  functional  requirements  described  in  the  business  case 
are  satisfied  and  to  determine  the  operational  effectiveness  and  suitability  of  the  increment.  The 
functional  sponsor,  informed  by  lOT&E  results  and  DOT&E  recommendations  (for  DBSs  on  OSD  T&E 
oversight),  issues  a  written  declaration  that  the  system  has  achieved  IOC.  The  limited  fielding  phase 
ends  when  phase  requirements  are  satisfied,  lOT&E  is  complete,  and  IOC  is  declared. 

14.4.6  Full  Deployment  Phase 

The  purpose  of  the  full  deployment  phase  is  to  field  an  increment  of  capability  for  operational  use  in 
accordance  with  the  business  case.  Entrance  criteria  for  this  phase  include  the  completion  of  lOT&E  or 
other  required  testing,  declaration  of  IOC,  and  satisfaction  of  the  DOTMLPF  solution  outlined  in  the 
business  case.  The  full  deployment  phase  begins  at  the  FDD  when  the  MDA  reviews  the  business  case, 
the  lOT&E  results  and  DOT&E  recommendations  (for  DBSs  on  OSD  T&E  oversight),  and  other 
requirements  to  determine  whether  the  capability  is  ready  to  proceed  to  full  deployment.  The  MDA 
decision  is  documented  in  an  ADM. 

14.4.7  O&S  Phase 

The  purpose  of  the  O&S  phase  is  to  execute  a  support  program  that  meets  materiel  readiness  and 
operational  support  performance  requirements  and  sustains  the  system  in  the  most  cost-effective 
manner  over  its  total  life  cycle.  Planning  for  this  phase  begins  prior  to  program  initiation  and  is 
summarized  in  the  business  case.  O&S  has  two  major  efforts:  life  cycle  sustainment  and  disposal.  The 
O&S  phase  begins  when  an  increment  or  DBS  has  been  fully  deployed.  A  close-out  review  is  conducted 
during  the  O&S  phase  to  solicit  user  feedback  and  enable  understanding  of  how  well  a  recently 
completed  increment  meets  the  needs  of  users  before  finalizing  the  requirements  for  subsequent 
increment(s).  A  close-out  review  is  held  with  the  responsible  IRB  and  includes  a  post-implementation 
review  report.  At  the  end  of  its  useful  life,  an  increment  is  disposed  of  in  accordance  with  all  statutory 
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and  regulatory  requirements  and  policy  including  but  not  limited  to  those  relating  to  safety,  security, 
and  the  environment. 

14.5  Summary 

DBS  investments  need  to  be  managed  with  the  kind  of  acquisition  management  rigor  and  discipline  that 
is  embodied  in  relevant  guidance  and  best  practices  so  that  each  investment  will  deliver  expected 
benefits  and  capabilities  on  time  and  within  budget. 
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Test  &  Evaluation  Management  Guide 

Chapter  15  -  Software  and  IT  Testing  Considerations 

15.1  Introduction 

Software  development  has  historically  been  a  major  risk  for  many  DoD  systems.  This  risk  has  been  true 
for  software  embedded  as  part  of  a  military  weapon  system  as  well  as  for  that  found  in  other  AISs,  such 
as  personnel  records  management,  financial  accounting,  logistics,  etc.  Although  testing  for  so-called 
software-intensive  systems  follows  the  same  general  T&E  precepts  outlined  elsewhere  in  this  guide, 
certain  characteristics  of  the  software  development  effort  are  important  to  understand.  This  chapter 
discusses  these  characteristics,  provides  an  overview  of  key  parts  of  a  typical  software  development 
process  and  the  associated  leverage  points  for  more  effective  testing. 

Software  T&E  encompasses  the  entire  system  development  life  cycle,  from  concept  and  requirements 
development  to  design,  operations,  maintenance,  and  upgrade/retirement.  Difficulty  in  adequately 
testing  software  and  the  associated  software  quality  problems  are  not  limited  to  DoD  or  the  Federal 
Government.  AISs,  once  developed  and  tested  in  the  host  hardware  that  is  commonly  off-the-shelf,  are 
essentially  ready  for  fielding.  On  the  other  hand,  software  in  weapon  systems,  once  integrated  in  the 
host  hardware,  continues  to  be  tested  as  a  component  of  the  subsystem(s)  and  ultimately  the  total 
system  and  is  not  ready  until  the  total  system  has  successfully  demonstrated  required  performance  and 
has  been  completely  integrated.  Any  changes  to  the  hardware  configuration  may  require  software 
changes.  Sometimes  hardware  performance  deficiencies  can  be  solved  by  modification  to  the 
embedded  software,  potentially  causing  late-stage  integration  problems  and  the  need  for  retesting. 
Additionally,  much  embedded  weapons  system  software  is  safety-critical,  real-time  software  operating 
under  severe  timing  constraints.  A  system  is  real-time  if  the  total  correctness  of  an  operation  depends 
not  only  upon  its  logical  correctness,  but  also  upon  the  time  in  which  it  is  performed.  The  classical 
conception  is  that  the  completion  of  an  operation  after  its  deadline  is  due  is  considered  useless- 
ultimately,  this  delay  may  cause  a  critical  failure  of  the  complete  system. 

15.2  Role  Of  The  Software  Specification  Review 

Software  considerations  are  evaluated  at  several  key  milestones  in  the  development  process,  including 
the  Software  Design  Review,  PDR,  and  CDR.  The  development  of  all  software  systems  involves  a  series  of 
development  activities  that  are  driven  by  an  underlying  SE  paradigm  that  does  not  change  for  software. 
However,  because  software  itself  is  an  invisible,  highly  flexible,  and  easily  changed  product  with  many 
ways  to  define  quality,  without  careful  considerations  of  these  unique  characteristics,  frequent 
opportunities  exist  for  the  introduction  and  insertion  of  errors. 

Errors  and  defects  may  occur  at  the  inception  of  the  process  when  the  requirements  may  be 
erroneously  specified  or  later  in  the  development  cycle  when  integration  is  being  done.  Figure  15-1 
illustrates  how  software  errors  can  accumulate  over  the  development  process,  generating  downstream 
errors  (avalanche),  and  why  early  detection  of  them  is  critical.  The  so-called  error  avalanche  can  include 
untested  functions,  unrepresentative  environments,  and  insufficient  repetitions,  incorrect  expected 
results,  and  misinterpreted  results. 
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Figure  15-1  "Error  Avalanche"  Showing  How  Software  Errors  Can  Accumulate 

Numerous  studies  have  shown  that  many  software  errors  originate  in  the  requirements  stages  of  the 
contractor's  software  development  process,  yet  little  effort  is  devoted  to  finding  and  correcting 
software  errors  at  this  stage,  when  the  cost  of  correction  is  cheapest.  Figure  15-2  shows  the  distribution 
summary  of  software  errors  across  the  lifecycle  software  development  activities. 

Frequently,  the  Government  does  not  have  good  insight  into  this  early  stage  of  the  development 
process-but  they  should!  Tester  participation  in  the  SSR  or  equivalent  review,  which  occurs  after 
software  requirements  analysis  for  the  to-be-developed  system  is  complete,  is  highly  recommended. 


Lifecycle  SW  Development  Activity 

Initial  $  Spent 

Errors  Introduced 

Errors  Found 

Relative  Cost 

Requirements  Analysis 

5% 

55% 

18% 

1.0 

Design  Activities 

25% 

30% 

10% 

1.0-1 .5 

T  esting  Activities 

60% 

10% 

50% 

1. 5-5.0 

Documentation 

10% 

— 

— 

— 

Software  Support 

— 

5% 

22% 

10-100 

Figure  15-2  Software  Error  Distribution  Summary  (Notional) 

The  SSR  is  a  technical  review  held  at  a  subsystem  level  that  is  unique  to  software.  It  is  completed  prior  to 
the  software  PDR.  At  the  SSR,  software  requirements  are  defined,  ultimately  becoming  the  functional 
baseline  for  software.  As  part  of  this  review,  test  acceptance  criteria  for  each  software  requirement  are 
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established.  Typically,  an  SSR  is  held  for  each  major  software  item  or  software  configuration  item  (SCI)  or 
group  of  SCIs  comprising  a  subsystem.  An  SCI  or  a  CSCI  (older  terminology  still  in  use)  is  a  collective  set 
of  software  routines  and  programs,  under  configuration  management,  that  accomplishes  a  well-defined, 
cohesive  goal  (a  mission  planning  CSCI,  a  human-computer  interface  CSCI,  an  avionics  CSCI,  etc).  The 
allocation  of  top-level  requirements  to  them  and  their  determination  are  one  of  the  many  outputs  of 
the  overall  SE  process. 

Some  of  the  key  questions  addressed  at  the  SSR  that  are  important  from  a  tester's  perspective  include 
the  following: 

•  Is  a  requirements  verification  and  test  matrix  complete? 

•  As  stated,  are  all  requirements,  include  derived  ones,  "testable"? 

•  Are  all  proposed  software  test  events  traceable  to  system  requirements? 

•  Is  software  that  requires  safety  or  other  certification  scheduled  to  be  certified? 

•  Has  an  early  Operational  Assessment  been  considered  for  software? 

•  Are  sufficient  resources  provided  as  part  of  the  developer's  software  test  plan? 

•  Does  the  proposed  test  schedule  have  provisions  for  re-work? 

•  Are  there  sufficient  test  assets,  including  resources  and  infrastructure,  programmed  for  later 
testing? 

•  Have  Software  Engineering  Environment  key  resources  been  identified? 

•  Is  all  required  support  software  going  to  be  available  when  required  for  testing? 

•  Is  the  TEMP  aligned  with  projected  software  testing  activities? 

The  completion  of  software  requirements  analysis  and  the  SSR  (or  similar  review)  establishes  the 
allocated  baseline  for  the  software  for  a  particular  SCI  or  set  of  Sis.  This  allocated  baseline  is  then  used 
for  subsequent  software  design  and  development.  Typically,  this  baseline  is  formally  documented  in  a 
software  requirements  specification  (SRS)  and  an  associated  interface  requirements  specification  (IRS) 
or  their  equivalents.  Key  to  the  tester  is  the  part  of  these  documents  that  contains,  generally  as  an 
annex,  a  requirements  traceability  matrix  (RTM).  The  RTM  should  show  each  software  requirement 
including  derived  requirements,  their  traces  to  higher  level  requirements  and  how  each  will  be 
validated,  and  how  the  specifications  are  verified  (e.g.,  inspection,  analysis,  demonstration,  or  test). 

Derived  requirements  result  from  the  analysis  of  system-level  requirements  done  as  part  of  the 
software  requirements  analysis.  They  are  not  explicitly  stated  in  a  higher  level  document  but  are  needed 
to  flesh  out  the  software  design.  Software  is  especially  rich  in  derived  requirements,  which  if  not 
carefully  controlled  and  planned  for,  can  add  unexpected  costs  and  schedule  to  the  program.  Indeed, 
the  original  rationale  by  the  USAF  Studies  Board  in  the  1980s  for  use  of  the  SSR  was  based  on  analysis  of 
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a  number  of  failed  programs  in  which  details  of  the  complete  scope  of  the  derived  software 
requirements  was  not  understood  until  later  in  the  program  when  extensive  rework  was  needed. 

It  is  important  from  a  tester's  perspective  that  completion  of  the  SSR  also  initiates  more  detailed  test 
planning  parallel  activities  on  the  part  of  the  developer.  These  culminate  in  SCI/CSCI  test(s)  that  will 
qualify,  via  a  software  qualification  test  (SQT),  the  software  as  correct.  Correct  in  this  sense  is  that  it 
meets  its  baselined  requirements  as  specified  by  the  SRS/IRS  RTM.  After  SQT,  the  SCI/CSCI  is  ready  to  be 
integrated  into  a  larger  subsystem.  These  activities,  which  occur  after  the  SSR,  vary  by  development 
approach/contractor  but  generally  happen  as  shown  in  Figure  15-3. 


BEST  PRACTICE  State  software 
requirements  in  measurable,  testable  terms 
«  •  »  and  specify  methods  for  evaluation  and 
testing  as  early  In  the  lifecycle  as  possible 
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Figure  153  Illustrative  Software  Test  Planning  Activities 
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15.3  Role  Of  The  TRR  For  Software 

The  TRR,  previously  discussed  at  the  system  level  in  Chapter  1 ,  is  a  multi-disciplined  technical  review 
designed  to  ensure  readiness  to  proceed  into  formal  test.  As  stated  in  Reference  (j) ,  the  TRR  assesses 
test  objectives,  test  methods  and  procedures,  scope  of  tests,  and  safety  and  confirms  that  required  test 
resources  have  been  properly  identified  and  coordinated  to  support  planned  tests.  The  TRR  verifies  the 
traceability  of  planned  tests  to  program  requirements  and  user  needs.  It  determines  the  completeness 
of  test  procedures  and  their  compliance  with  test  plans  and  descriptions.  The  TRR  also  assesses  the 
system  under  review  for  development  maturity,  cost/schedule  effectiveness,  and  risk  to  determine 
readiness  to  proceed  to  formal  testing. 

Historically,  much  like  the  SSR,  the  TRR  was  originally  developed  as  a  review  dealing  with  readiness  to 
start  SQT.  Over  the  years,  it  was  recognized  that  a  formal  review  of  readiness  for  the  start  of  testing  is  a 
universal  best  practice  for  all  parts  of  a  system  under  development  and  so  its  use  has  expanded  to 
where  the  TRR  is  generally  considered  to  be  a  formal  gating  event  to  determine  readiness  for  systems- 
level  testing.  However,  its  use  as  part  of  the  software  development  process  is  still  valid  and  highly 
recommended.  A  software  TRR  helps  testers  gain  visibility  of  potential  software  test  problems  and 
correct  them  earlier  in  the  development  life  cycle  before  software  items  get  integrated  into  larger 
subsystem(s)  or  even  into  the  system,  where  rework  can  become  prohibitively  more  expensive.  Some  of 
the  key  items  typically  examined  at  software  TRR  can  include: 

Requirements  Changes  .  Any  changes  to  the  SRS  or  IRS  or  other  software  allocated  baseline  documents 
that  have  been  approved  since  SSR  and  that  impact  testing. 

Design  Changes  .  Any  changes  to  the  software  design  document  that  have  been  made  since  PDR  and 
CDR  and  that  impact  software  testing. 

Software  Test  Plans  and  Descriptions  .  Any  changes  to  previously  approved  software  test  plans  and 
software  test  descriptions. 

Software  Test  Procedures  .  Procedures  to  be  used  in  conducting  testing,  including  retest  procedures  for 
test  anomalies  and  corrections. 

Software  Unit  Testing  .  Software  unit  integration  test  cases  and  procedures  used  in  conducting  prior 
software  unit  tests  and  other  lower  level  tests  and  the  test  results.  Closure  status  on  software  problem 
reports. 

Software  Test  Resources  and  Tools  .  Status  of  the  development  facility  hardware,  any  Government- 
furnished  software  or  GFE  essential  to  success  of  the  test,  availability  of  test  personnel,  and  supporting 
test  software  and  materials,  including  software  test  tools  used,  and  their  qualification  and  certification  if 
required. 

Requirements  Traceability  .  The  traceability  between  requirements  and  their  associated  tests. 
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Software  Measures  .  Status  of  the  program  based  on  relevant  software  measures.  Examples  include 
breadth  and  depth  of  testing,  requirement  test  levels,  problem  report  status,  actual  versus  planned 
staffing  profiles,  and  software  complexity. 

Dry  Run  Results  .  Results  from  a  dry  run  of  planned  procedures. 

15.4  Software  Development  Process 

Generally,  for  DoD  safety-critical  systems,  software  development  follows  a  classic  top-down  approach 
consisting  of  overlapping  phases  of  requirement  analysis,  architecture/preliminary  design,  detailed 
design,  coding,  and  a  series  of  tests  beginning  with  software  unit  testing  followed  by  integration  and 
associated  qualification  testing.  Qualified  SCIs  are  integrated  into  subsystems  and  finally  into  the 
complete  system. 

Typically,  a  software  unit  is  considered  to  be  the  smallest  independently  testable  component  of 
software  making  up  an  SCI.  Software  units  are  low-level  routines  and  programs.  An  SCI  may  be 
composed  of  hundreds  of  software  units  that  have  to  be  tested  and  integrated. 

These  stages  of  development  are  separated  by  various  lower  level  reviews  for  the  software  and  at  the 
subsystems  and  system  level  that  examine  development  progress  and  assess  risks.  For  safety-critical 
systems,  analysis  of  the  effort  will  also  include  formal  evaluation  of  various  work  products  in  accordance 
with  MIL-STD-882E  (Reference  (as)),  provisions  of  which  are  generally  contractually  imposed  as 
appropriate  for  these  types  of  DoD  systems. 

These  activities,  displayed  in  the  context  of  a  notional  system-level  WBS,  are  illustrated  in  Figure  15-4. 
Not  shown  are  the  various  subsystem-  and  system-level  reviews  (PDRs,  CDRs,  etc.)  that  occur  at  higher 
levels  in  the  WBS  as  the  system  is  integrated.  When  the  various  activities  are  drawn  in  this  manner,  the 
diagram  mimics  a  V  and  sometimes  is  referred  to  as  a  V-diagram.  V-diagrams  are  a  common  way  of 
depicting  SE  processes  as  well. 
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Figure  15-4  Illustrative  Software  Development  Activities  in  System  WBS  Context 
15.5  The  Potential  Power  Of  Human-Based  Testing 

As  shown  in  Figure  15-2,  software  errors  generated  during  the  software  requirements  analysis  phase  are 
the  most  difficult  to  find  and  the  most  costly  to  fix.  This  late  discovery  phenomenon  leads  to  more 
expensive  corrections  later  on,  exacerbated  at  all  stages  of  development  by  the  software  error 
avalanche  depicted  in  Figure  15-1.  The  obvious  solution,  therefore,  is  to  prevent  those  errors  from  being 
inserted  into  the  system  in  the  first  place,  which  would  minimize  the  amount  of  discovered  errors 
and/or  needed  rework  later  on.  However,  software  testing  in  the  classic  sense  of  running  software  on 
the  target  computer  does  not  occur  until  the  upward  leg  of  the  Figure  15-4  V-diagram.  What  can  be 
done  to  improve  this  state  of  affairs? 

One  way  to  improve  this  situation  is  to  use  human-based  testing.  Human-based  testing  occurs  on  the 
downward  leg  of  the  software  V-diagram.  The  earliest  stages  of  software  development  are  characterized 
by  heavy  human  involvement  in  basic  requirements  analysis,  design,  and  coding  processes.  Human- 
based  testing  (sometimes  called  static  testing)  encompasses  non-computer-based  methods  of 
evaluating  documentation  and  discovering  errors  covering  such  areas  as  software  requirements. 
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architectures,  designs,  interfaces,  and  code.  Human-based  tests  are  performed  multiple  times  as 
software  development  proceeds  and  matures  and  are  typically  performed  as  part  of  requirements 
analysis,  design,  and  early  coding  stages  of  development. 

Several  activities  are  encompassed  by  the  term  human-based  testing.  These  activities  include  the 
following,  which  are  listed  in  order  of  increasing  rigor  and  effectiveness: 

•  Desk  Checking  .  A  self-evaluation  is  made  by  programmers  of  their  own  work.  There  is  a  low 
probability  of  identifying  their  errors  of  logic  or  coding. 

•  Walk-  through  .  A  group  of  peers  evaluates  the  work  of  others  and  gives  direct  feedback  to  the 
authors.  Roles  can  be  shared  among  the  participants.  The  walk-through  leader  or  the  author 
may  serve  as  the  recorder.  The  walk-though  leader  may  be  the  author. 

•  Formal  Inspection  .  Consists  of  two  to  six  participants  (including  the  author).  An  inspection  is  a 
highly  disciplined  effort  led  by  an  impartial  trained  facilitator  trained  in  inspection  techniques. 
Determination  of  remedial  or  investigative  action  for  a  discovered  anomaly  is  a  mandatory 
element,  although  this  resolution  does  not  normally  occur  in  the  actual  inspection.  Per  IEEE  Std 
1028-2008  (Reference  (at)),  collection  of  data  for  the  purpose  of  analysis  and  improvement  of 
software  engineering  procedures  is  a  mandatory  element  of  software  inspections. 

Initially  developed  and  proofed  many  years  ago  at  IBM  Federal  Systems  Division,  formal  inspections  are 
fairly  straightforward  and  simple,  and  if  properly  done,  can  be  highly  effective  in  eliminating  errors  early 
in  the  development  process. 

Although  walk-throughs  have  the  potential  of  30  percent  to  40  percent  early  defect  removal,  repeated 
studies  have  shown  that  formal  inspections,  properly  done,  have  a  defect  removal  rate  on  the  order  of 
70  percent.  The  results  mean  the  size  of  the  error  avalanche  is  a  lot  smaller  and  makes  later,  computer- 
based  testing  more  effective.  Use  of  formal  inspections  is  strongly  recommended  as  a  contractor  best 
practice. 

Although  Government  testers  are  not  normally  part  of  the  contractor's  formal  inspection  team,  they 
should  determine  whether  the  contractor  is  actually  using  such  a  process  properly  and  gain  insight  into 
its  effectiveness.  Its  use  (or  lack  of  use)  can  give  testers  early  insights  into  possible  later  software  testing 
risks  and  associated  schedule  slips  due  to  unforeseen  rework. 

15.6  "Black  Box"  Versus  "White  Box"  Testing 

Software  computer-based  testing  involves  using  a  combination  of  black  box  and  white  box  testing. 

These  terms  are  defined  as  follows: 

•  Black  Box  Testing  .  Functional  testing  of  a  software  unit  without  knowledge  of  how  the  internal 
structure  or  logic  will  process  the  input  to  obtain  the  specified  output.  Within-boundary  and 
out-of-boundary  stimulants  test  the  software's  ability  to  handle  abnormal  events  and  to 
produce  the  expected  outputs  given  appropriate  inputs.  The  most  likely  cases  are  tested  to 
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provide  a  reasonable  assurance  that  the  software  will  demonstrate  specified  performance.  Most 
computer-based  testing  after  software  unit  tests  consists  of  are  black  box  tests. 

•  White  Box  Testing  .  Structural  testing  with  insight  into  the  internal  logic  and  software  structure. 
This  process  provides  an  opportunity  for  more  extensive  identification  and  testing  of  critical 
paths.  Test  scripts  are  developed  with  an  understanding  of  the  internal  structure  of  the  software 
under  test.  Software  unit  testing  is  performed  as  a  white  box  test. 

Testing,  as  illustrated  by  the  upward  portion  of  the  V-diagram  in  Figure  15-4,  should  be  performed  from 
the  bottom  up.  The  smallest  controlled  software  modules-software  units-are  tested  individually  via 
white  box  tests,  and  discovered  errors  are  resolved  at  the  lowest  possible  level  in  the  system  structure. 
Modules  are  then  combined  or  integrated  and  tested  in  larger  aggregate  groups  or  builds.  When  this 
process  is  complete,  the  software  item  (SCI)  is  tested  in  its  entirety  and,  after  a  successful  TRR, 
ultimately  qualified. 

Although  the  Government  tester  is  not  directly  involved  in  the  white  box  testing  at  the  software  unit 
level,  gaining  visibility  into  how  well  this  testing  was  performed  will  give  good  insights  into  later  test 
problems.  Frequently,  when  the  baseline  schedule  for  a  program  is  compressed,  in  order  to  maintain 
key  schedule  dates  (one  of  which  is  typically  the  SQT,  noted  in  Figure  15-3),  software  unit  testing  suffers 
and  is  not  adequately  performed  by  the  developer.  The  result  can  be  that  these  undiscovered  errors 
remain  until  later  black  box  integration  testing,  when  their  removal  is  more  expensive  and  requires 
rework  of  code  at  the  software  unit  level. 

15.7  The  Impossibility  of  Exhaustive  Software  Testing 

A  truism  in  the  software  testing  community  states  that  software  testing  can  only  indicate  the  presence 
of  errors,  not  their  absence.  This  statement  is  true  because  even  the  simplest  software  designs  rapidly 
exceed  the  capacity  to  test  all  possible  alternatives. 

For  instance,  in  the  power  distribution  panel  example  from  Combinatorial  Testing  by  Rick  Kuhn 
(Reference  (au)),  depicted  in  Figure  15-5,  there  is  a  set  of  34  power  distribution  switches  (two  rows  of  17 
switches).  A  software  program  monitors  the  positions  of  these  two-way  switches  to  determine 
appropriate  actions.  Because  this  is  a  safety-critical  system,  suppose  we  wanted  to  exhaustively  test  all 
possible  switch  combinations  to  absolutely  ensure  safe  operation.  Flow  many  individual  test  cases  would 
be  required  to  be  run?  The  number  of  individual  test  cases  required  would  be  quite  large. 

Because  the  switches  are  two-position  switches  and  there  are  34  of  them,  the  total  individual  test  cases 
required  would  be  17,179,869,184  1.7  x  10^° tests.  There  are  around  31  million  seconds  in  a  year, 
and  assuming  each  test  takes  1  second  to  manually  physically  set  the  switches,  etc.,  the  time  for  a 
complete  exhaustive  test  would  be  1.7  x  10^°/31  x  10®  548  years.  Clearly,  exhaustive  testing  for  this  or 
any  other  nontrivial  software  system  is  out  of  the  question! 

Flowever,  various  methods  and  engineering  techniques  can  be  used  to  improve  test  efficiency  and  gain 
confidence  in  the  limited  scope  of  software  testing  that  can  be  performed  given  typical  schedule 
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constraints.  These  include  such  techniques  as  combinatorial  or  pairwise  testing,  done  within  the  context 
of  DOE,  which  maximizes  the  utility  of  each  test  case. 


Figure  15-5  Testing  Example 

Additionally,  many  other  test  tools  exist  that  typically  are  part  of  the  developer's  software  engineering 
environment  and  can  dramatically  improve  productivity  of  software  testing.  Some  of  these  types  of 
tools  can  automatically  parse  proposed  test  cases  and  evaluate  the  breadth  and  depth  of  testing  to 
ensure  that  every  program  branch  and  line  of  code  is  tested  at  least  once.  Others  evaluate  the  logical 
complexity  of  the  code  and  identify  areas  in  which  the  software  could  be  simplified  to  improve  its 
correctness  and  maintainability.  Other  design  tools  are  used  to  plan  software  testing  activities  and  can 
generate  test  artifacts  that  drive  later  testing  activities. 

Although  the  operation  and  use  of  these  types  of  tools  is  beyond  the  scope  of  this  chapter,  the 
Government  tester  should  nevertheless  be  aware  of  their  use.  If  employed  effectively,  they  are  a  way  to 
increase  the  effectiveness  of  the  developers  computer-based  testing.  Their  absence  or  improper  use  in  a 
developer's  organization  is  a  risk  to  the  testing  effort.  This  is  why  an  assessment  of  these  types  of  tools 
should  be  one  of  the  review  items  at  a  software  TRR. 

15.8  Software  Error  Categorization  And  Priority 

Not  all  discovered  software  errors  are  equal.  Some  have  significant  functional  and  operational  impacts 
whereas  others  are  in  the  realm  of  suggested  improvements  and  cosmetic  engineering.  Although 
currently  there  is  no  DoD  policy  as  such  mandating  how  errors  are  classified,  at  one  time  a  priority 
scheme  was  mandated  by  a  now-retired  DoD  software  development  military  standard. 

This  was  a  simple,  workable  classification  scheme  that  stood  the  test  of  time.  It  subsequently  appeared 
as  a  suggested  methodology  in  various  replacement  commercial  standards  as  well.  This  software  error 
priority  classification  scheme  is  illustrated  in  Figure  15-6. 
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Example 

Classification 

Schemes 

Priority  Classification  Scheme 

Priority  1:  Prevents  Mission  Accomplishment  or 
jeopardizes  safety  or  other  "critical"  requirement 
Priority  2:  Adversely  affect  Mission  Accomplishment  or 
cost,  schedule,  performance  or  software  support  +  no 
work-around  exists 

Priority  3:  Adversely  affect  Mission  Accomplishment  or 
cost,  schedule,  perfonnance  or  software  support  +  a 
work-around  exists 

Priority  4:  User/Operator  or  support  inconvenience 
Priority  5:  All  other  problems 


Category  Classifications 

•  Project  Plans 

•  Operational  Concept 

•  System/Software  Requirements 

•  Design 

•  Code 

•  Databases/data  files 

•  Test  Plans,  Descriptions,  Reports 

•  User,  Operator  or  Support  Manuals 

•  Other  Software  Products 


Figure  15-6  Priority  Classification  Scheme  for  Software  Errors 

As  software  errors  are  discovered,  it  is  important  from  a  process  improvement  perspective  to  attempt 
to  identify  at  what  point  in  the  development  process  they  were  made.  Therefore,  many  developers  also 
categorize  the  source  where  or  how  the  error  was  introduced  into  the  development  process  with  a  view 
toward  improving  that  part  of  the  process.  The  data  in  Figure  15-2  showing  software  error  distributions 
was  derived  from  such  analyses  performed  over  many  projects. 

Keep  in  mind  that  Figure  15-6  is  generic.  One  suggested  best  practice  that  some  programs  have  used 
quite  successfully  is  to  tailor  the  priority  classification  categories  with  respect  to  the  actual  system 
domain  in  coordination  with  end  users.  For  example,  for  a  medical  system,  the  priority  classifications 
would  be  restated  in  terms  of  medical  issues;  for  an  avionics  system,  they  would  be  restated  with 
respect  to  flight  safety,  etc.  Issues  occasionally  arise  at  scoring  conferences  in  which  software  problems 
are  being  adjudicated  into  one  of  these  five  priorities;  restatement  of  priorities  into  operational  terms 
before  that  stage  can  eliminate  a  lot  of  these  issues. 

15.9  Overview  of  Software  Measurement  With  T&E  Applications 

Because  software  is  an  invisible  product  with  many  ways  to  define  its  quality  ilities  (suitability,  reliability, 
maintainability,  etc.)  and  many  ways  to  measure  them,  it  was  realized  early  on  in  the  DoD  that  a  formal 
way  to  measure  and  gauge  software  development  progress  and  associated  quality  was  needed.  Over 
many  years,  the  discipline  of  software  measurement  has  been  developed  and  refined;  and  used 
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appropriately,  software  measurement  is  an  essential  way  to  get  visibility  into  potential  development, 
quality,  and  testing  issues. 

One  of  the  DoD  SE  technical  management  core  processes  is  technical  assessment.  Technical  assessment 
consists  of  three  interrelated  activities:  (1)  EVM,  (2)  Technical  reviews  and  audits,  and  (3)  TPM.  Software 
measurement  is  a  specific  subset  of  TPM  that  is  tailored  to  the  unique  characteristics  of  the  software 
product. 

Software  measures  are  risk  and  issue  driven.  Given  a  set  of  identified  risks  and  issues,  a  set  of  metrics 
are  developed  that  quantify  and  measure  the  impact  of  those  identified  risks  and  issues.  Typically, 
software  measures  are  plotted  and  formatted  as  various  curves.  Assessment  of  the  shape  of  the  curves 
and  their  changes  over  time  can  give  good  insights  to  predicting  potential  later  problems.  Ideally,  action 
can  be  taken  early  enough  before  Brooks  Law  takes  effect. 

Brooks  Law  states  that  adding  staff  to  a  late  software  project  makes  it  later.  It  was  coined  by  Fred 
Brooks  in  his  1975  book.  The  Mythical  Man-Month.  Two  factors  are  at  work:  (1)  It  takes  some  time  for 
the  people  added  to  a  project  to  become  productive;  they  must  become  educated  about  the  work,  and 
this  education  requires  diverting  resources;  and  (2)  Communication  overheads  increase  as  the  number 
of  people  (or  subcontractors)  increases.  The  number  of  different  communication  channels  increases 
along  with  the  square  of  the  number  of  people;  so,  for  example,  doubling  the  number  of  people  on  a 
project  results  in  four  times  as  many  different  conversations.  Although  there  are  exceptions  to  Brooks 
Law,  it  has  generally  held  true  for  many  programs. 

Some  relevant  software  measures  of  interest  to  testers  include  requirements  stability,  computer 
resource  utilization,  complexity,  test  progress,  test  resource  utilization,  problem  report  status,  problem 
report  aging,  problem  report  priority,  and  depth  of  testing.  Some  of  these  measures  are  briefly 
summarized  in  sections  15.9.115.9.7. 

Much  of  the  discussion  and  examples  in  this  section  have  been  derived  from  handbooks  published  by 
the  Practical  Software  and  Systems  Measurement  organization.  The  Practical  Software  and  Systems 
Measurement  project  is  a  DoD  effort  that  has  developed  tools  and  handbooks  concerning  software 
measurement.  The  Practical  Software  and  Systems  Measurement  Website  at  http://www.psmsc.com/  is 
an  excellent  resource. 

15.9.1  Requirements  Stability 

Requirements  stability  measures  changes  in  requirements  over  time,  as  shown  in  Figure  15-7.  If 
software  requirements  are  still  changing  when  the  developer  is  in  detailed  design  or  even  worse,  during 
coding,  it  is  a  problem.  Each  requirement  should  be  baselined  by  the  SSR;  if  not,  the  associated  RTM 
(with  testing  criteria)  may  need  to  be  re-baselined  before  testing. 
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Requirements  Stability 


■  ror^ 


Figure  15-7  Example  of  Requirements  Stability  Measure 
15.9.2  Computer  Resource  Utilization 

Computer  resource  utilization  measures  the  utilization  of  some  capacity  of  the  underlying  hardware 
(e.g.,  processing  power,  memory  size,  bus  transfer  rates)  that  is  deemed  critical  to  the  performance  of 
the  system,  as  shown  in  Figure  15-8. 
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CPU  utilization 


Figure  15-8  Example  of  Computer  Resource  Utilization  Measure 

This  measure  is  important  because  as  reserve  capacity  is  approached,  software  performance  may  be 
degraded  and  provisions  for  adding  new  functionality  in  subsequent  increments  may  be  compromised. 
Typically,  initial  thresholds  are  set  at  a  lower  level  (shown  in  Figure  15-8  as  50  percent)  to  allow  for  this 
type  of  growth  and  evolution. 

15.9.3  Test  Progress 

Test  progress  measures  progress  of  low-level  software  unit  testing  against  the  plan.  In  Figure  15-9,  the 
simple  linear  extrapolation  of  the  passed  line  against  the  original  planned  schedule  indicates  a 
significant  slip  until  the  projected  90  tests  are  completed. 
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Test  Progress 

Components  Successfully  Tested 


□  Plan 
♦  Attempted 
A  Passed 


Figure  15-9  Test  Progress  Measure  Indicating  Schedule  Slip 

Because  these  software  units  collectively  make  up  an  SCI  that  will  undergo  later  qualification  testing,  the 
significant  slip  in  low-level  software  unit  testing  is  indicative  of  a  substantial  slip  in  that  test.  If  that  SCI 
has  critical  functionality,  the  subsequent  DT&E  and  even  the  start  of  OT&E  may  be  impacted  as  well. 

15.9.4  Test  Resource  Utilization 

As  depicted  in  Figure  15-10,  this  measure  compares  planned  usage  of  test  resources  with  actual  usage. 

In  this  case,  the  actual  amount  of  test  resource  usage  is  less  than  projected.  This  difference  may  be  due 
to  several  possibilities:  (1)  The  software  is  of  better  quality  than  anticipated  and  fewer  errors  are  being 
found,  (2)  The  software  test  cases  are  deficient  in  terms  of  depth  and  breadth  of  testing  and  therefore 
are  being  finished  too  quickly,  and/or  (3)  Not  enough  testers  are  available  to  run  the  planned  tests  due 
to  staffing  shortfalls. 

The  measure  itself  does  not  provide  the  root  cause,  and  further  investigation  is  warranted.  If  the  test 
facility  is  of  a  special  nature  (e.g.,  a  secure  facility  or  one  that  is  specially  instrumented)  and  thus  is  not 
readily  available  except  on  a  long-lead  time  scheduling  basis,  this  underutilization  could  also  indicate  a 
future  potential  schedule  slip. 
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Test  Facilities 
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Figure  15-10  Test  Facility  Underutilization  Example 
15.9.5  Problem  Report  Status 

This  measure  compares  reported  software  problem  reports  with  closed  problem  reports.  Typically, 
these  types  of  reports  are  segregated  by  software  error  priority.  The  example  in  Figure  15-11  shows  a 
troubled  project  with  many  priority  1  errors,  and  at  the  current  closure  rate  by  extrapolation,  there  is  a 
looming  (around  4  months)  schedule  slip  until  all  priority  1  errors  are  closed  out. 
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Problem  Report  Status 


■  R»port«d 

A  CtosAd 

Priority 
1  Errors 


Figure  15-11  Problem  Report  Closure  Rate  Status  Problems 

Regression  testing  is  conducted  to  close  problem  reports.  The  number  of  problem  reports  will  stabilize 
and  eventually  decrease  for  well-designed  software. 

15.9.6  Problem  Report  Aging 

This  measure  keeps  track  of  the  time  taken  to  clear  problem  reports.  Generally,  there  will  be  a  target 
closure  time,  shown  in  Figure  15-12  as  less  than  8  weeks.  This  rate  is  one  of  the  assumptions  the 
developer  makes,  for  rework  time,  and  is  factored  in  as  part  of  initial  scoping  of  the  length  of  the  overall 
development  schedule. 

In  this  case,  the  average  time  being  taken  to  clear  problem  reports  is  higher  than  anticipated.  This  may 
indicate  an  underlying  staffing  problem  (not  enough  personnel  to  fix  discovered  problems)  or  rework 
taking  longer  than  anticipated  due  to  lack  of  test  resources. 

At  any  rate,  it  is  likely  there  will  be  some  impact  to  the  overall  schedule  including  any  follow-on  tests 
that  may  have  been  programmed.  If  this  situation  persists,  the  overall  development  schedule  may 
require  restructuring. 


181 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcms 


/KI-L0JJwi 


t 

o 

I 

E 

§ 

I 


Problem  Report  Aging 
Open  Problem  Reports 


100-r 


eo-- 


60-- 


40-- 


20-- 


Avwog«  =  57  HTML'S 
rarg«f  <  0 


<  1  1-2  3-4  5-8 


9-18  >18  weeks 


Weeks  Open 

Prefect:  PSM  Oats  as  of  31  Jan  00 


Figure  15-12  Problem  Report  Aging  Example 
15.9.7  Problem  Report  Status  -  Open  by  Priority 

This  measure  rank  orders  the  open  problem  reports  by  priority,  as  shown  in  Figure  15-13.  Over  time, 
you  would  expect  to  see  the  developer's  efforts  focused  on  the  high-priority  problem  reports  and  that 
the  number  of  those  reports  would  decrease  over  time.  If  not,  it  may  indicate  that  the  developer  is 
working  problems  that  are  easy  to  fix  instead  of  the  important  ones.  Also,  it  may  indicate  that  the 
developer  is  reclassifying  high-priority  problem  reports  to  lower  categories.  If  this  reclassification  is 
being  done  without  Government  knowledge,  that  is  a  potential  problem  as  well. 

Figure  15-14  shows  an  alternative  way  of  tracking  problem  report  severity  over  time.  The  columns  show 
the  number  of  defects  that  have  been  open  for  a  given  time  period  by  severity,  with  the  values  in 
parentheses  reflecting  the  status  from  the  previous  reporting  period.  By  using  this  format  the  reader  can 
see  the  aging  by  severity  as  well  as  closure  rate  (current  versus  prior  period)  without  needing  multiple 
charts  or  sources. 
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Figure  15-13  Problem  Report  Open  Status  Example 


Assigned  and  Submitted  Defects  -  Days  Open 

Severity 

0-30 

31-60 

61-90 

91-120 

>120 

1 

9  (18) 

12(3) 

1  (1) 

2(3) 

4(2) 

2 

92  (99) 

41  (28) 

13  (11) 

5(8) 

19  (18) 

3 

48  (45) 

6(4) 

8  (11) 

3(0) 

16  (18) 

4 

16  (15) 

3(3) 

3(3) 

1  (1) 

4(4) 

5 

0(1) 

5(4) 

0(1) 

3(4) 

4(3) 

Total 

165  (178) 

67  (42) 

25  (27) 

14  (16) 

47  (45) 

Figure  15-14  Tracking  Problem  Report  Severity 
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Some  Service  test  agencies  have  lists  of  desirable  measures  and  procedures  for  measurement  that  are 
recommended  within  their  Service  to  use  as  a  tool  for  software  testing.  Also,  although  some  developers 
may  use  some  sort  of  software  measurement  program  internally,  software  metrics  in  general  and  those 
appropriate  for  software  testing  must  be  contractually  specified  with  Government  access  provisions. 

This  section  only  briefly  summarized  the  discipline  of  software  measurement.  For  more  information, 
many  online  software  measurement  training  resources  are  available  through  DAU. 

15.10  Independent  Verification  and  Validation  (IV  &  V) 

IV&V  is  a  process  that  is  appropriate  for  some  categories  of  systems.  Per  IEEE  Standard  1012  (Reference 
(av)),  the  three  components  of  IV&V  can  be  defined  as  follows: 

•  Independent .  Independent  means  that  the  process  is  independent,  both  managerially  and 
financially,  from  the  developer  whose  activities  are  being  evaluated  and  that  the  staff  members 
performing  IV&V  tasks  are  not  the  same  staff  members  as  those  doing  development  tasks. 

•  Verification  .  In  the  context  of  IV&V,  verification  is  defined  as  the  process  of  evaluating  software 
at  the  end  of  the  software  development  process  to  ensure  compliance  with  software 
requirements. 

•  Validation  .  In  the  context  of  IV&V,  validation  is  defined  as  the  process  of  determining  whether 
the  products  of  a  given  phase  of  the  software  development  cycle  fulfill  the  requirements  for  that 
cycle  established  in  the  previous  phase. 

NOTE:  The  terms  verification  and  validation  as  used  in  the  context  of  IV&V  are  different  from  the  V&V 
technical  processes  that  are  part  of  the  DoD  SE  process  set  and  are  different  from  the  use  of  these  terms 
in  the  VV&A  process  described  in  Chapter  18  for  M&S. 

Although  developers  perform  V&V  as  an  inherent  part  of  their  development  process,  for  high-risk 
systems  (e.g.,  safety-critical  systems  or  systems  processing  classified  data)  in  which  the  investment  can 
be  justified,  these  processes  are  repeated  and  performed  by  an  independent  agent  working  directly  for 
the  Government.  That  agent  is  the  IV&V  contractor. 

For  DoD  safety-critical  systems,  the  IV&V  agent  typically  works  within  the  processes  specified  in 
Reference  (as) .  Other  services  that  are  performed  by  IV&V  contractors  can  vary  widely  depending  on 
the  needs  of  the  program  and  generally  include  some  of  the  following: 

•  Management  Support  Services  .  These  services  include  reviews  of  program/project  plans, 
schedules,  milestones,  and  proposals  as  well  as  management  plans  such  as  quality  assurance 
and  configuration  management.  The  goal  is  to  identify  inconsistencies,  errors,  or  other  issues 
that  may  present  a  risk  to  the  program/project,  providing  management  with  an  early 
identification  of  potential  problems. 
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•  Engineering  Support  Services  .  These  services  include  evaluation  of  the  adequacy  and 
completeness  of  individual  technical/development  products  to  ensure  that  their  content 
conforms  to  applicable  standards,  contractual  provisions,  or  other  requirements. 

•  Test  Support  Services  .  These  services  include  evaluation  of  test  documentation  such  as  plans, 
test  cases,  and  test  procedures  to  ensure  that  their  content  conforms  to  applicable  standards, 
contractual  provisions,  or  other  requirements.  The  evaluation  would  verify  that  the  test 
documents  correctly  link  to  each  other  and  correctly  reflect  the  life  cycle  needs,  security, 
performance,  and  other  requirements  based  on  the  level  of  testing.  Additionally,  the  IV&V  agent 
may  witness  the  actual  testing  performed. 

•  Technical  Review  and  Audits  Support  Services  .  The  IV&V  agent  often  provides  input  concerning 
the  adequacy  of  project  artifacts  or  life  cycle  processes,  in  the  form  of  reports  or  presentations, 
in  support  of  technical  reviews  and  audits. 

Special  contractual  provisions  must  be  imposed  that  allow  the  IV&V  agent  to  access  developer 
materials,  facilities,  working  documents,  etc. 

15.11  T&E  Issues  Associated  With  Spiral  and  Agile  Development  Approaches 
15.11.1  Spiral  Model 

Spiral  development  is  a  risk-driven,  prototype-based,  cyclical,  iterative  build-test-fix-test-deploy 
software  development  process.  It  can  help  evolve  the  software  product  to  better  meet  user  needs. 

Spiral  development,  properly  applied,  can  provide  a  variety  of  benefits  including  the  following: 

•  Facilitate  changes  to  software  requirements  that  are  defined  by  operational  mission  needs, 
technology  opportunities,  experimentation  results,  and  technology  obsolescence. 

•  Support  a  continuous  process  of  realistic  integrated  testing  that  evolves  over  time  to  measure 
the  operational  effectiveness,  suitability,  and  supportability  of  each  increment  of  the  software¬ 
intensive  system. 

•  Improve  the  visibility  of  product  development  baselines  to  allow  early  inputs  via  prototyping 
from  the  systems  user. 

The  spiral  development  process  may  also  impact  the  software  measurement  process  by  changing  the 
priorities  of  the  information  needs  of  a  project  as  the  product  evolves.  Spiral  development  usually 
increases  the  number  of  changes  that  will  be  made  to  the  requirements  and  systems  design  baselines, 
expanding  the  need  for  configuration  control  and  related  measures.  The  spiral  model  also  has  other 
potential  problems  if  not  carefully  controlled.  Mismanagement  of  the  prototyping  effort  can  drive  up 
costs.  Because  it  is  risk  driven,  an  inconsistent  or  weak  risk  management  process  that  does  not  openly 
identify  risks  on  the  part  of  either  the  developer  or  the  acquirer  will  dramatically  reduce  the  chance  of 
program  success.  Strong  configuration  management  and  a  flexible  contracting  approach  are  essential. 
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Because  the  product  in  spiral  development  evolves  continuously,  testing  can  be  a  challenge.  Careful 
coordination  with  the  development  team  and  strict  baseline  control  for  interim  products  are  essential. 
Additionally,  because  spiral  development  uses  prototypes  extensively  as  risk  mitigation  approaches  to 
better  understand  requirements  before  the  complete  system  is  built,  knowledge  of  whether  these 
prototypes  will  be  reused  in  the  final  system  or  recoded  will  drive  the  amount  of  regression  testing 
necessary.  Spiral  development  is  generally  not  appropriate  for  safety-critical  systems  in  which  the 
provisions  of  Reference  (as)  must  be  applied. 

15.11.2  Agile  Development  Methods 

Agile  software  development  refers  to  a  group  of  software  development  methodologies  based  on 
iterative  and  incremental  development  in  which  requirements  and  solutions  evolve  through 
collaboration  between  self-organizing,  cross-functional  teams.  Agile  development  is  being  embraced  by 
segments  of  the  DoD  as  a  potentially  effective  approach  for  software  development  under  the  right 
circumstances  for  some  categories  of  software-intensive  systems. 

Terms  such  as  scrum,  extreme  programming,  and  crystal  clear  are  also  sometimes  used  to  refer  to  agile 
approaches.  They  are  characterized  by  their  proponents  as  lightweight  software  development  methods. 
So-called  lightweight  software  development  methods  evolved  in  the  mid-1990s  as  a  reaction  against 
heavyweight  methods,  which  were  characterized  by  their  critics  as  heavily  regulated,  regimented, 
micromanaged  approaches  to  software  development. 

Definitions  of  what  characterizes  agile  development  have  evolved  over  time  as  developers  have  gained 
experience  with  these  methods.  Because  of  ambiguity  in  defining  agile  development,  its  adherents  have 
outlined  its  key  principles  via  the  so-called  Agile  Manifesto,  reproduced  in  Figure  15-15. 


Figure  15-15  Spiral  Model  Software  Development  Approach 
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Agile  development  releases  products  in  small,  frequent  releases,  and  testing  of  these  small  releases  is 
performed  incrementally  as  part  of  the  release  cycle.  Close  user  coordination  is  key,  and  user 
participation  as  a  member  of  the  agile  development  team  is  essential  for  success.  Issues  remain 
regarding  scalability  to  large  systems,  work  scope,  contracting  approaches,  efficiency,  requirements 
creep,  and  risk  management. 

15.11.3  Agile  Development  and  Testing 

Testing  within  an  agile  development  model  has  many  challenges  including  the  following:  (1)  there  are 
many  variations  of  agile  development,  all  with  the  goal  of  frequent  deliveries  and  shorter  development 
times;  (2)  regression  testing  in  the  classic  sense  may  be  difficult  because  of  the  flux  in  baselines;  (3)  the 
development  process  itself  encourages  requirements  changes  as  the  product  evolves  in  order  to  better 
meet  the  need  of  the  end  users  and  limit  the  need  for  post-fielding  rework;  (4)  agile  development  is  not 
appropriate  for  all  systems. 

DoD  testing  for  agile  development  will  evolve  as  more  experience  is  gained  with  the  process  itself.  Given 
the  wide  range  of  possibilities  falling  under  the  agile  development  umbrella,  it  is  likely  that  each  test 
program  will  require  tailoring  by  the  tester  to  get  the  best  return  for  the  T&E  investment.  Integrated 
testing  should  be  an  essential  part  of  the  testing  approach. 

The  DoD  is  becoming  interested  in  using  agile  approaches  as  a  method  to  help  improve  IT  system 
development  outcomes  for  certain  categories  of  AISs.  One  suggested  agile  life  cycle  that  incorporates 
testing  is  depicted  in  Figure  15-16. 
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Continuous 
Integration  . 

and  rigorous  ( • 

Agile  Testing 


Voice  o1  the  User 


•  Testing  is  a  shared  resource 

—  Risk-based,  mission  focused 
—  One  team:  DT,  Of,  interoperability,  security 
—  Reciprocity 

•  Continuous  user  involvement 

•  Federate  DoD  capabilities;  leverage  virtual  environments 


Note:  Disgramderivedfrom  “IT  Acquis'tion  Reform"  briefing,  DoO  IT  Acquisition  Taskforce 


Figure  15-16  Notional  Agile  Development  Model  Depicting  Testing 
15.12  COTS  Considerations 

DoD  policies  require  development  of  an  appropriate  T&E  strategy  for  CIs  that  includes  evaluating 
potential  CIs  in  a  system  test  bed,  when  practical;  focusing  test  beds  on  high-risk  items;  and  testing  Cl 
upgrades  for  unanticipated  side  effects  in  areas  such  as  security,  safety,  reliability,  and  performance. 

Many  DoD  systems  employ  COTS  software.  The  amount  used  typically  varies  by  domain,  with  weapons 
systems  generally  using  fewer  COTS  products  than  AISs.  Indeed,  some  AISs  may  be  implemented  as  a  so- 
called  enterprise  resource  planning  (ERP)  system,  in  which  the  complete  functionally  is  provided  by  a 
tailored  COTS  product.  An  ERP  system  is  a  collection  of  software  programs  that  ties  together  an 
enterprise's  various  functions,  such  as  human  resources,  finance,  logistics,  etc. 

Properly  used  and  understood,  COTS  software  can  have  great  advantages  in  certain  application  settings. 
At  one  time,  the  extensive  use  of  COTS  software  in  DoD  systems  was  felt  by  some  to  be  a  universal 
panacea  for  DoD  software  development  problems;  however,  numerous  lessons  learned  from  less-than- 
successful  COTS  implementations  showed  otherwise.  Some  of  these  lessons  learned  include  the 
following: 
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•  Unit  level  testing/component  testing  is  generally  impossible. 

•  Documentation  of  many  COTS  products  is  not  complete  or  robust  or  incorrect. 

•  Complex,  non-standard  interfaces,  which  may  not  be  well-defined,  abound  in  COTS  products. 

•  Market  leverage  may  not  exist  to  force  vendor  bug  fixes  for  DoD  unique  needs. 

•  Component  understanding  depends  mostly  on  vendor  claims. 

•  Formal  requirements  documents  as  such  are  generally  not  available. 

•  Current  COTS  use  in  the  DoD  environment  may  not  match  its  original  design  environment  in  the 
commercial  market  place. 

•  Real-time  performance  may  be  marginal. 

•  Robustness  and  reliability  of  many  COTS  products  are  generally  lower  when  compared  with  DoD 
custom  code. 

•  Higher  COTS  use  implies  more  difficult  system  level  integration  testing  due  to  the  number  of 
applications  that  have  to  be  integrated  and  associated  "glue  code".  The  number  of  interfaces 
that  have  to  be  integrated  grows  as  the  square  of  the  number  of  COTS  products  to  be 
integrated.  Frequently,  this  integration  is  done  by  using  custom  software  euphemistically  called 
glue  code. 

•  "Evaluation"  of  product  suitability  occurs  long  before  formal  testing. 

•  Frequent  market-driven  releases  complicate  regression  testing. 

•  Vendor  configuration  management  of  COTS  products  is  frequently  poorly  done  and  can  lead  to 
conflicting  baselines  in  a  system. 

15.13  Mature  Software 

Typically,  test  plans  cite  the  need  for  mature  software  as  a  criterion  to  move  forward  (typically  for 
OT&E)  in  the  testing  process  but  never  specifically  define  the  term.  Obviously,  this  is  a  program-specific 
criterion.  To  be  effective,  such  criteria  have  to  be  multidimensional,  encompassing  a  range  of  measures 
such  as  software  problems,  functionality,  configuration  control,  and  management. 

One  set  of  suggested  software  maturity  criteria  (developed  many  years  ago  by  the  DoD,  in  response  to 
several  OT&E  deficiencies  cited  by  the  Government  Accountability  Office)  per  the  DOT&E  memorandum 
(Reference  (aw))  is  provided  for  consideration  in  Figure  15-17. 
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Software  Problems 


0  No  Priority  I  or  II  problems 
0  Impact  Analysis  for  Priority  III 

0  Functionality  available  before  OT&E 
0  DT  completed 

0  External  interfaces  functionally  certified 

0  PMOs  ID  unmet  critical  parameters 
0  Acquisition  Executive  (+  Operational 
Tester  must  agree)  must  certify  that: 

•  SW requirements  design  stable 

•  Depth  S  breadth  of  testing  is  OK 

•  DT  has  demo’d  required  functions 

0  Deficiency  reporting  process  in  place 
0  Software  CM  system  in  place 
0  System  is  baselined 
0  Test  Agency  has  access  to  CM  system 


0AII  changes  completed 


Figure  15-17  Example  of  Software  "Maturity"  Criteria 

Other  related  considerations  can  be  found  in  the  DOT&E  memorandum  (Reference  (ax)).  This  document 
presents  a  set  of  guidelines  for  tailoring  pre-deployment  test  events  to  the  operational  risk  of  a  specific 
system  increment  being  acquired  under  OSD  oversight. 

15.14  Summary 

There  is  a  growing  body  of  software  testing  technologies  that  can  be  applied  to  testing  of  AIS  and 
weapon  system  software.  Systematic  T&E  techniques  are  far  superior  to  ad  hoc  testing.  Implementation 
of  an  effective  software  T&E  plan  requires  a  set  of  strong  technical  and  management  controls,  effective 
use  of  human-based  testing  techniques,  close  coordination  with  the  developers  test  activities,  metrics, 
and  an  understanding  of  how  the  unique  nature  of  the  software  product  impacts  the  testing  process. 
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Test  &  Evaluation  Management  Guide 
Chapter  16  -  Testing  for  Vulnerability  and  Lethality 

16.1  Introduction 

This  chapter  addresses  the  assessment  of  the  vulnerability  and  lethality  aspects  of  a  system  via  T&E 
practices  and  procedures.  In  particular,  this  chapter  describes  the  legislatively  mandated  LFT  program, 
which  has  been  established  to  evaluate  the  survivability  and  lethality  of  developing  systems.  It  also 
discusses  the  role  of  T&E  in  assessing  a  systems  ability  to  perform  in  a  nuclear  combat  environment.  The 
discussion  of  testing  for  nuclear  survivability  is  based  primarily  on  information  contained  in  Appendix  E 
of  the  Nuclear  Matters  Handbook  (Reference  (ay)).  The  handbook,  located  at  the  Deputy  Assistant 
Secretary  of  Defense  for  Nuclear  Matters  website,  is  intended  to  be  an  unofficial  reference  and  is, 
therefore,  neither  authoritative  nor  directive  . 

16.2  LFT 

16.2.1  Historical  Background 

In  March  1984,  OSD  chartered  a  joint  T&E  program  designated  The  Joint  Live-Fire  Program.  This  program 
was  to  assess  the  vulnerabilities  and  lethality  of  selected  U.S.  and  threat  systems  already  fielded.  The 
controversy  over  joint  LFT  of  the  Army's  Bradley  Fighting  Vehicle  System,  subsequent  congressional 
hearings,  and  media  exposure  resulted  in  provisions  being  incorporated  in  the  National  Defense 
Authorization  Act  (NDAA)  of  FY  1987  (Reference  (az)).  Specifically,  this  law  stipulates  that  major  program 
(ACAT  I  and  II)  development  may  not  proceed  beyond  LRIP  until  realistic  survivability  or  (in  the  case  of 
missiles  and  munitions)  lethality  testing  has  been  completed.  This  act  requires  an  OSD-managed  LFT 
program  for  major  acquisition  programs  fitting  certain  criteria  (so-called  covered  systems). 

In  accordance  with  the  DOT&E  Memorandum  (Reference  (bb)),  the  term  covered  system  has  been 
clarified  as  a  system  that  the  DOT&E,  acting  for  the  SecDef,  has  determined  to  be  a  major  system  that  is 
(1)  user-occupied  and  designed  to  provide  some  degree  of  protection  to  its  occupants  in  combat,  or  (2) 
a  conventional  munitions  program  or  missile  program,  or  (3)  a  conventional  munitions  program  for 
which  more  than  1,000,000  rounds  are  planned  to  be  acquired,  or  (4)  a  modification  to  a  covered 
system  that  is  likely  to  affect  significantly  the  survivability  or  lethality  of  such  a  system,  or  (5)  any  other 
system  or  program  designated  for  LFT&E  oversight  by  the  DOT&E. 

The  SecDef  has  delegated  the  authority  to  waive  requirements  for  the  full-up,  system-level  LFT&E 
before  the  system  passes  the  program  initiation  milestone  to  the  USD(AT&L)  for  ACAT  ID  and  to  the  CAE 
for  ACAT  II  programs  when  LFT&E  would  be  unreasonably  expensive  and  impractical.  However,  in  that 
case,  an  alternate  LFT&E  program  must  still  be  accomplished.  Programs  subject  to  LFT  or  designated  for 
oversight  are  listed  on  the  OSD  annual  T&E  oversight  list. 

The  DoD  agent  for  management  of  the  LFT  program  is  the  DOT&E.  This  type  of  test  must  be  planned  to 
start  early  enough  in  the  development  process  to  impact  design  and  to  provide  timely  test  data  for  the 
OSD  LFT  report  required  for  the  FRPDR  and  interested  congressional  committees.  The  Service-detailed 
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LFT  plan  must  be  reviewed  and  approved  by  the  DOT&E,  and  LFT  must  be  addressed  in  Part  III  of  the 
program  TEMP. 

LFT  criteria  are  explicitly  specified  in  section  2366  of  Reference  (b).  These  provisions  are  summarized  in 
Figure  16-1. 


The  tenn  "eoreied  »yitem*  meant — 

(A) a  vehicle,  weapon  ptMfonn.  or  conventional  weapon  lyitem  that — 

(i)  iccludet  featurei  den^ned  to  provide  tome  degree  of  protection  to  utert  in  combac, 
and 

(ii)  It  a  major  tyilem  at  defined  in  Title  10  Section  2302  (5).  or 

(B)  any  other  tyttem  or  program  dengnaied  by  the  Secretary  of  Defence  for  purpotet  of  thit 
lection 

The  term  "major  muniiiont  program'  meant — 

(A)  a  munition  program  for  which  more  than  1.000.000  roundt  are  plarmed  to  be  accfuired.  or 

(B)  a  conventional  rounitioni  program  that  it  a  m^r  tyttem  within  the  meaning  of  that  term 
in  Title  10  Section  2302  (?) 


The  term  "realittic  turvivabihly  tetting*  meant,  in  the  cate  of  a  covered  tyttem  (or  a  covered 
product  improvement  program  for  a  covered  tyttem).  teituvg  for  vulneratabty  of  the  tyttem  m 
combat  by  finng  munitiont  likely  to  be  encountered  in  combat  (or  munitioni  with  a  capability 
timilar  to  Rich  munitioni)  at  the  tyttem  configured  for  combaC.  with  the  primary  emphant  on 
letting  vulnerability  with  retpecl  to  potential  uter  catualtiet  and  taking  into  equal  contideration  the 
tutceptibilily  to  attack  and  combat  performance  of  the  tyttem 


The  term  'realiitic  lethality  letting'  meant,  m  the  cate  of  a  major  munitioni  program  or  a  mitiile 
program  (or  a  covered  product  improvement  program  for  luch  a  program),  testing  for  lethality  by 
firing  the  munition  or  mitnle  concerned  at  appropriate  targeti  configured  for  combat 


The  term  'configured  for  combat',  with  respect  to  a  weapon  tyttem.  plaaform.  or  vehicle,  meant 
loaded  or  equipped  with  all  dangeroui  malenali  (mcluding  all  flamroablei  and  explonvei)  that 
would  normally  be  on  board  in  combat 


The  term  'covered  product  improvement  program'  meant  a  program  under  which — 

(A)  a  modification  or  upgrade  will  be  made  to  a  covered  tyttem  which  (at  determined  by  the 
Secretary  of  Defense)  it  likely  to  affect  significantly  the  survivability  of  such  tyttem.  or 

(B)  a  modification  or  upgrade  will  be  made  to  a  major  munitioni  program  or  a  mittile 
program  which  (at  determined  by  the  Secretary  of  Defense)  it  likely  to  affect  significantly 
the  lethality  of  themurution  or  misnle  produced  under  the  program 


Figure  16-1  Summary  of  Key  Parts  of  the  U.S.C.  Regarding  LFT 
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16.2.2  LFTs 

There  are  varying  types  of  LFTs.  The  matrix  in  Figure  16-2  illustrates  the  various  possible  combinations. 
Full-scale,  full-up  testing  is  usually  considered  to  be  the  most  realistic. 

The  importance  of  full-up  testing  has  been  well  demonstrated  by  the  joint  live-fire  (JLF)  tests.  Many 
examples  exist;  in  one  case,  these  tests  contradicted  earlier  conclusions  concerning  the  flammability  of  a 
new  hydraulic  fluid  used  in  F-15  and  F-16  aircraft.  Laboratory  tests  had  demonstrated  that  the  new  fluid 
was  less  flammable  than  the  standard  fluid.  However,  during  the  JLF  tests,  30  percent  of  the  shots  on 
the  new  fluid  resulted  in  fires  contrasted  with  15  percent  of  the  shots  on  the  standard  fluid. 

Although  much  insight  and  valuable  wisdom  are  to  be  obtained  through  the  testing  of  components  or 
subsystems,  some  phenomena  are  only  observable  when  full-up  systems  are  tested.  The  interaction  of 
such  phenomena  has  been  termed  cascading  damage.  Such  damage  is  a  result  of  the  synergistic  damage 
mechanisms  that  are  at  work  in  the  real  world  and  likely  to  be  found  during  actual  combat. 

LFT  provides  a  way  of  examining  the  damages  inflicted  not  only  on  materiel,  but  also  on  personnel.  For 
example,  per  the  DOT&E  Memorandum  (Reference  (bb)),  between  2007  and  2010,  extensive  ballistic 
testing  against  hard  body  armor  was  conducted  without  a  sound  test  operations  procedure.  At  the 
recommendation  of  the  DoD  Inspector  General,  the  DOT&E  developed  test  operations  procedures  for 
body  armor  ballistic  inserts  and  verified  that  the  procedures  were  implemented  DoD-wide. 

The  crew  casualty  problem  is  an  important  issue  that  the  LFT  program  is  addressing.  The  program 
provides  an  opportunity  to  assess  the  effects  of  the  complex  environments  that  crews  are  likely  to 
encounter  in  combat  (e.g.,  fire,  toxic  fumes,  blunt  injury  shock,  and  acoustic  injuries). 


Loading 

Full-up 

Inert* 

System 

Level 

Completes  system  configured  for  combat:  with 
combustibles  (e.g.,  personnel  carrier  tests, 
aircraft  "proof  tests) 

Complete  system:  no  combustibles  (e.g.  tests  of  new  armor  on 
actual  tanks,  aircraft  flight  control  tests) 

Subsystem 

Level 

Components,  subcomponents:  with 
combustibles  (e.g.,  fuel  cell  tests,  behind 
armor  penetration,  mock-up  aircraft,  engine 
fire  test) 

Components,  subcomponents,  structures,  terminal  ballistics, 
munitions  performance,  behind-armor  tests,  warhead 
characterization  (e.g.,  armor/warhead  interaction  tests,  aircraft 
component  structural  tests) 

*ln  some  cases,  targets  are  "seml-lnert,''  meaning  some  combustibles  are  on  board  but  not  all. 

(Example;  Tests  of  complete  tanks  with  fuel  and  hydraulic  fluid,  but  dummy  ammunition) 

Figure  16-2  Summary  of  Degrees  of  LFT 
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16.2.3  Role  of  M&S 

Survivability  and  lethality  assessments  have  traditionally  relied  largely  on  the  use  of  M&S  techniques. 
The  LFT  program  does  not  replace  the  need  for  such  techniques.  Such  predictions  are  useful  for  several 
reasons.  First,  they  assist  in  the  test  planning  process.  If  a  model  predicts  that  no  damage  will  be 
inflicted,  test  designers  and  planners  should  reexamine  the  selection  of  the  shot  lines  and/or  reassess 
the  accuracy  of  the  threat  representation.  Second,  pre-shot  model  predictions  provide  the  Services  with 
the  opportunity  to  validate  the  accuracy  of  the  models  by  comparing  them  with  actual  LFT  results.  At 
the  same  time,  the  LFT  program  reveals  areas  of  damage  that  may  be  absent  from  existing  models  and 
simulations.  Third,  pre-shot  model  predictions  can  be  used  to  help  conserve  scarce  target  resources. 

16.2.4  Illustrative  LFT  Process 

Plans  for  LFT&E  must  be  considered  as  part  of  test  planning  and  included  in  the  TEMP.  Figure  16-3  is  an 
illustrative  example  of  the  LFT&E  planning  and  execution  process.  Key  points  include  the  following: 

•  The  LFT&E  detailed  T&E  plan  is  the  basic  planning  document  used  by  OSD  and  the  Services  to 
plan,  review,  and  approve  LFT&E.  Services  will  submit  the  plan  to  the  DOT&E  for  comment  at 
least  30  days  before  test  initiation. 

•  The  LFT&E  plan  must  contain  general  information  on  the  systems  required  performance, 
operational  and  technical  characteristics,  critical  test  objectives,  and  the  evaluation  process. 

•  Each  LFT&E  plan  must  include  testing  of  complete  systems.  A  limited  set  of  LFTs  may  involve 
production  components  configured  as  a  subsystem  before  full-up  testing. 

•  A  Service  report  must  be  submitted  within  120  days  of  the  completion  of  the  LFT.  The  report 
must  include  the  firing  results,  test  conditions,  limitations,  and  conclusions  and  must  be 
submitted  in  classified  and  unclassified  form. 

•  Within  45  days  of  receipt  of  the  Service  report,  a  separate  LFT  report  will  be  produced  by  the 
DOT&E,  approved  by  the  SecDef,  and  transmitted  to  Congress.  The  conclusions  of  the  report  will 
be  independent  of  the  conclusions  of  the  Service  report.  Reporting  on  LFT&E  may  be  included  in 
the  weapon  systems  BLRIP  report  completed  by  the  DOT&E. 

•  Congress  shall  have  access  to  all  LFT  data  and  all  LFT  reports  held  by  or  produced  by  the 
Secretary  of  the  concerned  Service  or  by  OSD. 

•  The  costs  of  all  LFTs  shall  be  paid  from  funding  for  the  system  being  tested.  In  some  instances, 
the  Deputy  for  LFT&E,  DOT&E  may  elect  to  supplement  such  funds  for  the  acquisition  of  targets 
or  target  simulators,  although  the  ultimate  responsibility  rests  on  the  concerned  Service. 
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Figure  16-3  Illustrative  LFT&E  Planning  and  Execution  Process 

16.2.5  The  Role  of  the  Survivability/Vulnerability  Information  Analysis  Center  (SURVIAC) 

The  Survivability,  Vulnerability  Information  Analysis  Center  (SURVIAC)  is  the  DoD  focal  point  for 
nonnuclear  survivability/vulnerability  data,  information,  methodologies,  models,  and  analysis  related  to 
U.S.  and  foreign  aeronautical  and  surface  systems.  SURVIACs  role  is  to  collect,  analyze,  and  disseminate 
scientific  and  technical  information  related  to  all  aspects  of  survivability  and  lethality  for  aircraft,  ground 
vehicles,  ships,  and  spacecraft,  as  well  as  conventional  homeland  security  threats  including  chemical, 
biological,  directed-energy,  and  nonlethal  weapons. 

SURVIAC  is  a  contractor-operated  DoD  Information  Analysis  Center  sponsored  by  the  Defense  Technical 
Information  Center  (DTIC)  with  support  from  several  organizations  including  the  Joint  Aircraft 
Survivability  Program  Office  and  the  JTCG  on  Munitions  Effectiveness.  Among  other  available  services, 
SURVIAC  provides  the  following: 

•  An  extensive  library  of  relevant  survivability  and  lethality. 
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•  A  database  of  historical  combat  damage. 

•  Access  to  an  approved  set  of  models  and  simulations. 

16.3  Nuclear  Weapons  Testing 

The  testing  of  nuclear  weapons  is  highly  specialized  and  regulated.  PMs  involved  in  this  area  are  advised 
to  consult  authorities  within  their  chain  of  command  for  the  specific  directives,  instructions,  and 
regulations  that  apply  to  their  individual  situations.  Nuclear  weapons  tests  are  divided  into  categories  in 
which  the  responsibilities  of  the  Department  of  Energy,  the  Defense  Threat  Reduction  Agency  (DTRA), 
and  the  Military  Services  are  clearly  assigned.  The  Department  of  Energy  is  responsible  for  nuclear 
warhead  technical  tests;  DTRA  is  responsible  for  nuclear  weapons  effects  tests.  The  Services  are 
responsible  for  the  testing  of  Service-developed  components  of  nuclear  subsystems.  All  nuclear  tests  are 
conducted  within  the  provisions  of  the  Limited  Test  Ban  Treaty  that  generally  restricts  nuclear 
detonations  to  the  underground  environment.  Nuclear  weapons  testing  requires  extensive  coordination 
between  Service  and  Department  of  Energy  test  personnel. 

16.4  Testing  For  NH&S 
16.4.1  Background 

Nuclear  hardness  is  a  quantitative  description  of  the  physical  attributes  of  a  system  or  component  that 
will  allow  it  to  survive  in  a  given  nuclear  environment.  Nuclear  survivability  is  the  capability  of  a  system 
to  survive  in  a  nuclear  environment  and  to  accomplish  a  mission.  DoD  policy  requires  the  incorporation 
of  NH&S  features  in  the  design,  acquisition,  and  operation  of  major  and  non-major  systems  that  must 
perform  critical  missions  in  nuclear  conflicts.  Nuclear  hardness  levels  must  be  quantified  and  validated. 

The  T&E  techniques  used  to  assess  NH&S  include  nuclear  testing,  physical  testing  in  a  simulated 
environment,  M&S,  and  analysis.  Although  nuclear  tests  provide  a  high  degree  of  fidelity  and  valid 
results  for  survivability  evaluation,  they  are  not  practical  for  most  systems  because  of  cost,  long  lead 
times,  and  international  treaty  constraints. 

Underground  testing  is  available  only  on  a  prioritized  basis  for  critical  equipment  and  components  and  is 
subject  to  a  frequently  changing  test  schedule.  Physical  testing  provides  an  opportunity  to  observe 
personnel  and  equipment  in  a  simulated  nuclear  environment.  M&S,  and  analysis  are  particularly  useful 
in  the  early  stages  of  development  to  provide  early  projections  before  system  hardware  is  available. 
These  methods  are  also  used  to  furnish  assessments  in  an  area  that  cannot  be  directly  observed  through 
nuclear  or  physical  testing  because  of  safety  or  testing  limitations. 

Nuclear  survivability  must  be  incorporated  into  the  design,  acquisition,  and  operation  of  all  systems  that 
must  perform  critical  missions  in  a  nuclear  environment.  Nuclear  survivability  is  achieved  through  a 
combination  of  methods  such  as  hardness,  avoidance,  proliferation,  and  reconstitution.  Hardness  allows 
a  system  to  physically  withstand  a  nuclear  attack.  Avoidance  encompasses  measures  taken  to  avoid 
encountering  a  nuclear  environment.  Proliferation  involves  having  sufficient  systems  to  compensate  for 
probable  losses.  Reconstitution  includes  the  actions  taken  to  repair  or  resupply  damaged  units  in  time  to 
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complete  a  mission  satisfactorily.  There  are  many  possible  effects  from  a  nuclear  detonation,  including 
electromagnetic  pulse  (EMP),  ionizing  and  thermal  radiation,  blast,  shock,  dust,  debris,  blackout,  and  in 
some  cases,  transient  radiation  effects  on  electronics  (TREE).  TREE  is  the  damage  to  electronic 
components  by  initial  nuclear  radiation  gamma  rays  and  neutrons.  Each  weapon  system  is  susceptible  to 
some  but  not  all  of  these  effects.  Testers  must  identify  the  effects  that  may  have  an  impact  on  the 
system  under  development  and  manage  the  design,  development,  and  testing  of  the  system  in  a 
manner  that  minimizes  degradation.  The  distinction  between  nuclear  weapons  effects  survivability  and 
nuclear  weapons  system  survivability  is  shown  in  Figure  16-4. 


Nuclear  Weapons  Effects  Survivability 

Nuclear  Weapons  System  Survivability 

Survivability  of  Everything 

Survivability  of  Nuclear  Forces 

•  Nuclear  Weapons 

•  Nuclear  Weapons 

•  Nuclear  Force  Personnel 

•  Nuclear  Force  Personnel 

•  Nuclear  Force  Equipment 

•  Nuclear  Force  Equipment 

•  Conventional  Weapons 

•  Conventional  Force  Personnel 

•  Conventional  Equipment 

Against  the  Effects  of  Any  Threat: 

•  Nuclear  Weapons 

•  Chemical,  Biological  Weapons 

Against  the  Effects  of  Nuclear  Weapons 

•  Conventional  Weapons 

After  Figure  C.1,  Nuclear  matters  -  A  Practical  Guide 

•  Advanced  Technology  Weapons 

•  Special  Operations  Attacks 

•  Terrorist  Attacks 

Figure  16-4  Nuclear  Weapons  Effects  vs.  Systems  Survivability 

16.4.2  Assessing  Nuclear  Survivability  Throughout  the  System  Acquisition  Cycle 

The  PM  must  ensure  that  nuclear  survivability  issues  are  addressed  throughout  the  system  acquisition 
cycle.  During  assessment  of  concepts,  the  survivability  requirements  as  stated  in  the  Service  capability 
and  requirements  documents  should  be  verified,  refined,  or  further  defined.  During  the  systems  early 
design  stages,  trade-offs  between  hardness  levels  and  other  system  characteristics  (such  as  weight, 
decontaminability,  and  compatibility)  should  be  described  quantitatively.  Trade-offs  between  hardness, 
avoidance,  proliferation,  and  reconstitution  as  a  method  for  achieving  survivability  should  also  be 
considered  at  this  time.  During  advanced  engineering  development,  the  system  must  be  adequately 
tested  to  confirm  that  hardness  objectives,  criteria,  requirements,  and  specifications  are  met.  Plans  for 
NH&S  testing  should  be  addressed  in  the  TEMP.  The  appropriate  commands  must  make  provision  for 
test  and  hardness  surveillance  equipment  and  procedures  so  required  hardness  levels  can  be 
maintained  once  the  system  is  operational. 
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During  production,  deployment,  and  operational  support,  system  hardness  is  maintained  through  an 
active  hardness  assurance  program.  Such  a  program  ensures  that  the  end  product  conforms  to  hardness 
design  specifications  and  that  hardness  aspects  are  reevaluated  before  any  retrofit  changes  are  made  to 
existing  systems. 

Once  a  system  is  operational,  a  hardness  surveillance  program  may  be  implemented  to  maintain  system 
hardness  and  to  identify  any  further  evaluation,  testing,  or  retrofit  changes  required  to  ensure 
survivability.  A  hardness  surveillance  program  consists  of  a  set  of  scheduled  tests  and  inspections  to 
ensure  that  a  systems  designed  hardness  is  not  degraded  through  operational  use,  logistics  support, 
maintenance  actions,  or  natural  causes. 

16.4.3  NH&S  Test  Planning  Challenges 

The  following  challenges  are  associated  with  NH&S  testing: 

The  magnitude  and  range  of  effects  from  a  nuclear  burst  are  much  greater  than  those  from  conventional 
explosions  that  may  be  used  to  simulate  nuclear  bursts.  Nuclear  detonations  have  effects  not  found  in 
conventional  explosions.  The  intense  nuclear  radiation,  blast,  shock,  thermal,  and  EMP  fields  are  difficult 
to  simulate. 

The  yields  and  configurations  for  underground  testing  are  limited.  It  is  generally  not  possible  to  test  all 
relevant  effects  simultaneously  or  to  observe  possibly  important  synergism  between  effects. 

System-level  testing  for  nuclear  effects  is  normally  expensive,  takes  years  to  plan  and  conduct,  and 
requires  specialized  expertise.  Often,  classes  of  tests  conducted  early  in  the  program  are  not  repeated 
later.  Therefore,  operational  requirements  should  be  folded  into  these  tests  from  the  start,  often  early 
in  the  acquisition  process.  This  mandates  a  more  extensive,  combined  DT&E/OT&E  test  program  than 
normally  found  in  other  types  of  testing. 

PMs  and  TMs  must  remain  sensitive  to  the  ambiguities  involved  in  testing  for  nuclear  survivability.  For 
example,  there  is  no  universal  quantitative  measure  of  survivability,  and  statements  of  survivability  may 
lend  themselves  to  a  variety  of  interpretations.  Moreover,  it  can  be  difficult  to  combine  system 
vulnerability  estimates  for  various  nuclear  effects  into  an  assessment  of  overall  survivability. 

16.5  Summary 

The  survivability  and  lethality  aspects  of  certain  systems  must  be  evaluated  through  LFT.  These  tests  are 
used  to  provide  insights  into  the  weapon  systems  ability  to  continue  to  operate/fight  after  being  hit  by 
enemy  threat  systems.  Such  testing  provides  a  way  of  examining  the  damages  inflicted  not  only  on 
materiel,  but  also  on  personnel.  LFT  also  provides  an  opportunity  to  assess  the  effects  of  complex 
environments  that  crews  are  likely  to  encounter  in  combat.  Provisions  of  the  U.S.C.  apply  to  certain 
aspects  of  LFT. 

Nuclear  survivability  must  be  carefully  evaluated  during  the  system  acquisition  cycle.  Trade-offs 
between  hardness  levels  and  other  system  characteristics,  such  as  weight,  speed,  range,  cost,  etc.,  must 
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be  evaluated.  Nuclear  survivability  testing  is  difficult,  and  the  evaluation  of  test  results  may  result  in  a 
variety  of  interpretations.  Therefore,  PMs  must  exercise  caution  when  developing  test  objectives  related 
to  nuclear  survivability. 
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Test  &  Evaluation  Management  Guide 
Chapter  17  -  Logistics  Support  T&E 

17.1  Introduction 

Logistics  support  planning  for  materiel  acquisition  programs  begins  early  in  the  capabilities  assessments 
of  mission  needs,  continues  throughout  the  acquisition  life  cycle,  and  extends  past  the  deployment 
phase  into  the  O&S  phase.  To  be  most  effective,  logistics  support  test  and  evaluation  (LOG  T&E)  must 
mirror  this  timeline.  Careful  planning  and  execution  of  the  LOG  T&E  process  is  necessary  to  ensure  that 
the  RAM  objectives  of  the  system  are  adequately  identified  and  achieved.  Additionally,  the  importance 
of  the  PSMs  participation  in  integrating  key  aspects  of  the  LCSP  with  the  TEMP  is  essential.  Such 
coordination  will  ensure  that  LOG  T&E  objectives  are  properly  reflected  in  the  TEMP  and  that  adequate 
resources  are  available.  Figure  17-1  is  an  overview  of  the  different  aspects  of  LOG  T&E. 


Figure  17-1  Illustrative  Scope  of  LOG  T&E 

Logistics  system  support  planning  is  a  disciplined,  unified,  and  iterative  approach  to  the  management 
and  technical  activities  necessary  to  integrate  support  considerations  into  system  and  equipment 
design;  to  develop  support  requirements  that  are  related  consistently  to  readiness  objectives,  design, 
and  each  other;  to  acquire  the  required  support;  and  to  provide  the  required  support  during 
deployment  and  operational  support  at  affordable  cost.  LOG  T&E  planning  is  a  similar  disciplined 
approach  that  addresses  effective  T&E  design  to  assess  these  support  planning  considerations. 

A  key  part  of  the  LOG  T&E  planning  effort  occurs  at  the  T&E  WIPT  with  the  assignment  of  the  projects 
lead  logistician  to  that  team.  This  assignment  is  critical!  A  common  theme  among  programs  having 
difficulty  in  successfully  completing  reliability,  maintainability,  and  supportability  T&E  is  failure  to 
include  the  projects  logistician  in  the  test  planning  process,  or  involving  logisticians  too  late.  It  is 
important  to  engage  the  lead  design  engineers  during  the  design  phase  and  do  LOG  T&E  planning  early 
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so  that  reliability,  maintainability,  and  sustainability  T&E  can  be  performed  during  DT&E.  The  logisticians 
mantra  of  up  front  and  early  is  very  much  applicable  in  the  test  planning  phases  of  program 
development  to  accurately  forecast  readiness  and  cost  objectives  post-fielding. 

It  is  critical  to  include  the  designated  OTA  during  early  planning  to  lay  the  groundwork  for  future 
integrated  testing.  One  function  of  DT&E  is  to  perform  tests  that  provide  acquisition  personnel  with  an 
indication  of  how  the  system  will  fare  in  OT&E.  Bringing  in  the  OTA  early  to  develop  an  integrated  test 
schedule  will  save  time  and  conserve  resources  for  the  program. 

LOG  T&E  seeks  to  verify  and  validate  that  support  requirements  enhance  the  systems  design,  are  not  in 
conflict  with  one  another,  and  are  directly  related  to  readiness  objectives.  Early  planning  and  continued 
compliance  of  a  robust  LOG  T&E  program  will  ensure  that  the  required  support  is  available  during 
deployment  and  operational  support  at  an  affordable  cost. 

Figure  17-1  provides  an  overview  of  the  different  types  of  testing  involved  in  LOG  T&E.  Integrated 
testing  brings  in  OTAs  early  during  the  programs  test  planning  to  work  with  the  program  office  to  jointly 
design  tests  that  can  fulfill  data-gathering  requirements  for  both  DT&E  and  OT&E  (see  Chapter  8  of  this 
guide).  Integrated  testing  is  particularly  relevant  for  LOG  T&E  and  can  result  in  significant  savings  in  time 
and  other  resources. 

Logistic  support  can  be  categorized  into  integrated  product  support  (IPS)  elements  (referred  to  in  some 
older  documents  as  the  10  ILS  elements  but  subsequently  expanded  to  12),  as  illustrated  in  Figure  17-2. 
Per  Appendix  A  of  the  DoD  PSM  Guidebook  (Reference  (be)),  these  elements  are  evaluated  in  the 
supportability  T&E  phase  of  LOG  T&E. 
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INTEGRATED  PRODUCT  SUPPORT 
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Product  Support  Management 


Product  Support  is  enabled  by  Integrated  Product  Support  (IPS)  Elements  designed  to 
deliver  system  readiness  &  availability  while  optimizing  system  life  cycle  cost 


Figure  17-2  IPS  Elements 

The  IPS  elements  are  described  as  follows: 

•  Design  Interface  .  The  integration  of  the  quantitative  design  characteristics  of  SE  (reliability, 
maintainability,  etc.)  with  the  functional  logistics  elements  (i.e.,  IPS  elements).  These  design 
parameters  reflect  operational  suitability  requirements  and  specifically  relate  to  system 
requirements.  The  basic  items  that  need  to  be  considered  as  part  of  design  interface  include  the 
following: 

o  Reliability 

o  Maintainability 

o  Supportability 

o  IPS  elements 

o  Affordability 
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o  Configuration  management 
o  Safety  requirements 

o  Environmental  and  hazardous  material  requirements 
o  HSI 

o  Anti-tamper 
o  Habitability 
o  Disposal 
o  Legal  requirements 

•  Sustaining  Engineering  .  Involves  the  identification,  review,  assessment,  and  resolution  of 
deficiencies  throughout  a  systems  life  cycle.  Sustaining  engineering  returns  a  system  to  its 
baselined  configuration  and  capability  and  identifies  opportunities  for  performance  and 
capability  enhancement. 

•  Supply  Support .  Consists  of  all  management  actions,  procedures,  and  techniques  necessary  to 
determine  requirements  to  acquire,  catalog,  receive,  store,  transfer,  issue,  and  dispose  of 
spares,  repair  parts,  and  supplies.  This  means  having  the  right  spares,  repair  parts,  and  all 
classes  of  supplies  available  in  the  right  quantities,  at  the  right  place,  at  the  right  time,  and  at 
the  right  price. 

•  Maintenance  Planning  and  Management  .Establishes  maintenance  concepts  and  requirements 
for  the  life  of  the  system  for  both  hardware  and  software. 

•  Packaging,  Handling,  Storage,  and  Transportation  (PHS&T) .  The  combination  of  resources, 
processes,  procedures,  design,  considerations,  and  methods  to  ensure  that  all  systems, 
equipment,  and  support  items  are  preserved,  packaged,  handled,  and  transported  properly, 
including  environmental  considerations,  equipment  preservation  for  short  and  long  storage,  and 
transportability. 

•  Technical  Data  .  Identifies,  plans,  resources,  and  implements  management  actions  to  develop 
and  acquire  information  to: 

o  Operate,  install,  maintain,  and  train  on  the  equipment  to  maximize  its  effectiveness  and 
availability. 

o  Effectively  catalog  and  acquire  spare/repair  parts,  support  equipment,  and  all  classes  of 
supply. 

o  Define  the  configuration  baseline  of  the  system  to  effectively  support  the  Warfighter 
with  the  best  capability  at  the  time  it  is  needed. 
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•  Support  Equipment .  Consists  of  all  equipment  (mobile  or  fixed)  required  to  support  the 
operation  and  maintenance  of  a  system.  This  includes  but  is  not  limited  to  ground  handling  and 
maintenance  equipment,  trucks,  air  conditioners,  generators,  tools,  metrology  and  calibration 
equipment,  and  manual  and  automatic  test  equipment. 

•  Training  and  Training  Support .  Consists  of  the  policy;  processes;  procedures;  techniques; 
training  aids,  devices,  simulators,  and  simulations;  planning;  and  provisioning  for  the  training 
base  including  equipment  used  to  train  civilian  and  military  personnel  to  acquire,  operate, 
maintain,  and  support  a  system. 

•  Manpower  and  Personnel .  Involves  the  identification  and  acquisition  of  personnel  (military  and 
civilian)  with  the  skills  and  grades  required  to  operate,  maintain,  and  support  systems  over  their 
lifetime. 

•  Facilities  and  Infrastructure  .  Consists  of  the  permanent  and  semipermanent  real  property 
assets  required  to  support  a  system,  including  studies  to  define  types  of  facilities  or  facility 
improvements,  location,  space  needs,  environmental  and  security  requirements,  and 
equipment. 

•  Computer  Resources  .  Identify,  plan,  resource,  and  acquire  facilities,  hardware,  software, 
documentation,  manpower  and  personnel  necessary  for  planning  and  management  of  mission 
critical  computer  hardware  and  software  systems.  Includes  Automated  Logistics  Environment 
systems  (i.e..  Autonomic  Logistics  Information  System  of  the  Joint  Strike  Fighter),  hardened 
laptop  computers,  personal  electronic  digital  devices  used  in  electronic  technical  manuals,  etc. 

•  Product  Support  Management .  To  plan  and  manage  cost  and  performance  across  the  product 
support  value  chain,  from  design  through  disposal. 

17.2  Planning  For  Logistics  T&E 

17.2.1  Objectives  of  Logistics  T&E 

The  main  objective  of  LOG  T&E  is  to  verify  and  validate  that  the  R&M  predictions  as  specified  in  the 
design  documents,  user  requirements,  and  capabilities  documents  (CDD/CPD)  are  in  concert  with  the 
support  structure  (generally  considered  as  the  12  IPS  elements)  as  documented  in  the  LCSP.  Additional 
information  about  the  LCSP  can  be  found  in  Chapter  5  of  Reference  (j).  As  shown  in  Figure  17-1,  during 
DT,  LOG  T&E  consists  of  RAM  testing.  It  should  be  noted  that  R&M  testing  will  require  enough  test  hours 
to  validate  that  requirements  are  being  met  and  will  generally  continue  into  OT&E  or  until  the 
requirements  have  been  validated.  R&M  are  elements  of  design  and  drive  future  supportability 
requirements.  They  determine  what,  how  many,  and  how  much  of  each  of  the  12  IPS  elements  are 
required.  This  impact  is  measured  in  supportability  T&E  and  serves  as  verification  that  the  support 
structure  (the  12  IPS  elements),  as  defined  in  the  systems  LCSP,  is  designed  correctly. 

During  OT,  the  LOG  T&E  emphasis  shifts  from  evaluating  supportability  to  testing  and  evaluating  logistics 
suitability.  Whereas  supportability  measures  the  degree  to  which  system  design  characteristics  and 
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planned  logistics  resources  meet  system  peacetime  readiness  and  wartime  utilization  requirements, 
suitability  measures  the  degree  to  which  a  system  can  be  placed  and  sustained  satisfactorily  in  field  use. 
While  continuing  to  measure  the  impact  of  R&M  on  the  12  IPS  elements,  during  OT,  the  system  is 
typically  validated  against  standardized  suitability  COIs. 

COIs  are  key  operational  effectiveness  or  suitability  issues  that  must  be  examined  in  OT&E  to  determine 
the  systems  capability  to  perform  its  mission.  COIs  must  be  relevant  to  the  required  capabilities  and  of 
key  importance  to  the  system  being  operationally  effective,  operationally  suitable,  and  survivable,  and 
represent  a  significant  risk  if  not  satisfactorily  resolved.  A  COI  is  normally  phrased  as  a  question  that 
must  be  answered  in  the  affirmative  to  properly  evaluate  operational  effectiveness  (e.g..  Will  the  system 
detect  the  threat  in  a  combat  environment  at  adequate  range  to  allow  successful  engagement?)  and 
operational  suitability  (e.g..  Will  the  system  be  logistically  supportable?).  COIs  are  critical  elements  or 
operational  mission  objectives  that  must  be  examined.  COIs  are  broken  down  into  MOEs  (how  well  can 
the  system  perform  its  mission)  and  MOSs  (how  well  can  the  system  be  supported  in  the  field),  which  in 
turn  are  further  defined  by  MOPs. 

COI  examples  include  reliability,  availability,  compatibility,  transportability,  interoperability,  wartime 
usage  rates,  maintainability,  safety,  human  factors,  manpower  supportability,  logistics  supportability, 
documentation,  environmental  effects,  and  training  requirements.  Of  these,  the  COIs  directly  correlating 
to  the  12  IPS  elements  evaluated  in  DT  are  manpower  supportability,  logistics  supportability, 
documentation,  and  training  requirements. 

At  first  glance,  the  difference  between  LOG  T&E  DT  and  LOG  T&E  OT  seems  subtle.  In  reality,  the 
differences  are  much  greater.  Specifically: 

•  In  DT,  the  design  of  the  logistics  support  structure  is  measured  against  the  design  elements  of 
R&M.  Much  of  this  testing  is  performed  in  a  simulated  environment. 

•  In  OT,  the  design  of  the  logistics  support  structure  is  measured  on  a  fully  integrated  system 
against  the  CDD/CPD  design  elements  of  R&M  and  includes  operational  suitability  that  is 
evaluated  in  a  mission  context-real  environmental  conditions/locations  using  real  field 
maintainers-in  order  to  provide  meaningful  results.  Only  through  this  type  of  evaluation  can 
DoD  gain  insight  into  the  interactions  of  the  various  suitability  factors. 

Figure  17-3  shows  DT,  OT,  and  logistics  supportability  objectives  for  each  acquisition  phase. 
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Figure  17-3  Logistics  Supportability  Objectives  in  the  Overall  T&E  Program 
17.2.2  Logistics  Testability 

Based  upon  the  logistics  requirements  -  RAM  and  IPS  elements  (supportability)  -  both  stated  and 
derived,  the  PSM,  in  conjunction  with  the  project  lead  for  T&E,  develops  supportability  assessment 
plans.  These  plans  identify  the  testing  approach  and  the  evaluation  criteria  that  will  be  used  to  assess 
the  adequacy  of  supportability  and  design  requirements  and  associated  test  resources.  These  plans 
should  include  logistics  participation  in  the  assessment  of  the  mandatory  sustainment  KPP  for  materiel 
availability  and  KSAs  for  reliability  and  ownership  cost  requirements.  The  fact  that  the  availability  (or 
sustainment)  KPP  and  the  two  supporting  KSAs  (reliability  and  ownership  cost)  are  mandatory  is  not 
coincidental.  Achieving  high  reliability  has  a  cost,  as  does  achieving  superior  maintainability  and  high 
availability.  These  sustainment  metrics  taken  together  bound  the  trade  space  for  designing  in 
supportability  and  planning  affordable  life-cycle  product  support. 

The  logistics  support  manager  may  apply  the  best  practices  of  logistics  support  analysis  as  described  in 
Reference  (af) ,  portions  of  which  describe  how  to  develop  measurable  and  testable  supportability 
requirements.  Some  examples  of  suggested  testability  criteria  derived  from  the  handbook  are 
summarized  below: 
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•  Availability  Example  .  The  item  will  have  an  operational  availability  of  .95  measured  by  the  total 
operating  time  divided  by  the  sum  of  the  total  operation  time,  total  corrective  maintenance 
time,  total  preventive  maintenance  time,  and  the  total  administrative  and  logistics  downtime. 
The  vehicle  will  have  a  maintenance  ratio  of  the  total  scheduled  and  unscheduled  maintenance 
man-hours  per  hour  of  operation  (excluding  operator/crew  checks  and  daily  operating  service) 
that  does  not  exceed  the  following  values:  (1)  Organizational  0.140;  (2)  Direct  Support  0.043;  (3) 
Total  0.183. 

•  Operational  Availability  Example  .  The  system  (e.g.,  aircraft,  tank,  ship,  radio)  will  have  an 
operational  availability  of  completing  a  sustained  4-day  operation  using  only  onboard 
equipment  and  spares  without  resupply  or  support  from  personnel  other  than  the  operators. 

The  operational  test  of  the  system  will  be  used  to  verify  the  requirement  is  met.  The  test  will 
consist  of  two  systems  performing  four  each  of  Scenario  A,  as  identified  in  [enter  appropriate 
source  here],  and  two  each  of  Scenario  B  (surge),  as  identified  in  [enter  appropriate  source 
here].  Nine  of  the  12  scenarios  must  be  fully  executed  without  outside  resupply/assisted 
maintenance.  Additionally,  at  least  one  surge  scenario  must  be  completed  without  outside 
resupply/assisted  maintenance. 

•  Transportability  Example  .  The  vehicle  must  be  capable  of  being  rigged  for  air  drop  by  the  using 
unit  without  the  use  of  special  tools,  within  X  minutes.  The  vehicle  shall  be  capable  of  being 
sling-loaded  beneath  the  CH-47D  or  the  CH-53E  helicopters  using  integral  vehicle  lift  points. 

•  Maintainability  Example  .  Performance  of  duties  will  be  accomplished  by  Soldiers  within  the 
physical  capabilities  specified  in  [enter  appropriate  Service  regulation  here]  for  each  military 
occupational  specialty  designated  to  support,  operate,  maintain,  repair,  and  supervise  the 
employment  of  the  system.  Introduction  of  the  system  into  the  unit  inventory  will  not  cause  an 
increase  in  the  number  of  personnel  to  operate  or  support  personnel  in  excess  of  those 
currently  required. 

•  Materiel  Availability  Example  .  Materiel  availability  appears  to  be  very  similar  to  operational 
availability  (e.g.  (uptime)/(uptime+downtime));  however,  materiel  availability  is  calculated  for 
the  entire  fleet  (i.e.,  all  units  in  force  structure),  not  just  the  items/platforms  assigned  to  a 
particular  unit.  The  calculation  includes  assets  in  the  repair  pipeline,  at  depot  for  overhaul,  etc. 
The  numeric  value  for  materiel  availability  will  naturally  be  lower  than  the  numeric  value  for 
operational  availability,  but  it  will  portray  a  more  accurate  picture  of  total  cost  and  availability  of 
the  capability  to  the  force  structure. 

•  Reliability  Example  .  Reliability  may  initially  be  expressed  as  a  desired  failure-free  interval  that 
can  be  converted  to  a  failure  frequency  for  use  as  a  requirement  (e.g.,  95  percent  probability  of 
completing  a  12-hour  mission  free  from  mission-degrading  failure;  90  percent  probability  of 
completing  five  sorties  without  failure).  Specific  criteria  for  defining  operating  hours  and  failure 
criteria  must  be  provided  together  with  the  reliability . 
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17.2.3  TEMP 

Appropriate  LOG  T&E  information  should  be  included  in  the  TEMP.  Such  input,  derived  from  logistics 
supportability  planning,  should  include  specific  plans  for  testing  logistics  support,  descriptions  of 
required  operational  suitability,  and  required  testing  resources.  The  TEMP  is  the  primary  program 
document  for  test  resource  budgets  and  scheduling.  Therefore,  it  is  of  critical  importance  that  all  key 
test  resources  required  for  IPS  element  testing  (DT,  OT,  integrated  testing,  and  post-deployment 
supportability)  be  identified  in  the  TEMP. 

17.2.4  Planning  Guidelines  for  LOG  T&E 

Planning  guidelines  for  logistics  support  T&E  include  the  following: 

•  Develop  a  test  strategy  for  each  logistics  support-related  objective.  Ensure  that  DT&E, 
integrated  testing,  and  OT&E  planning  encompasses  all  IPS  elements.  The  general  objectives 
shown  in  Figure  17-3  must  be  translated  into  quantitative  and  qualitative  requirements  for  each 
acquisition  phase  and  each  program.  Quantitative  requirements  are  those  stated  as  KPPs  and 
KSAs  in  the  CDD/CPD,  whereas  qualitative  requirements  relate  to  the  12  IPS  elements. 

•  Incorporate  logistics  support  testing  requirements  into  the  formal  DT&E  and  OT&E  integrated 
testing  plans. 

•  Identify  logistics  support  T&E  that  will  be  performed  outside  of  the  normal  DT&E  and  OT&E. 
Include  subsystems  that  require  off-system  evaluation. 

•  Identify  all  required  resources,  including  test  articles  and  logistics  support  items  for  formal 
DT/OT/integrated  testing  and  separate  logistics  support  testing. 

•  Ensure  that  planned  T&E  will  provide  sufficient  data  on  high-cost  and  high-maintenance  burden 
items  (e.g.,  for  high-cost  critical  spares,  early  test  results  can  be  used  to  reevaluate  selection). 

•  Participate  early  and  effectively  in  the  TEMP  development  process  to  ensure  that  the  TEMP 
includes  critical  logistics  T&E  designated  test  funds  from  program  and  budget  documents. 

•  Identify  the  planned  utilization  of  all  data  collected  during  the  assessments  to  avoid 
mismatching  of  data  collection  and  information  requirements. 

•  Ensure  that  key  items  (e.g.,  parameters  to  be  measured,  methods  for  measurement  and  time 
frames,  and  any  penalties/incentives  associated  with  the  achieved  demonstrated  performance) 
are  well-defined  in  the  procurement  contract. 

•  Specify  in  the  TES  and  the  TEMP  how  reliability  will  be  tested  and  evaluated  during  the 
associated  acquisition  phase. 

•  Develop  RGCs  that  reflect  what  is  in  the  reliability  growth  strategy  and  employ  the  RGCs  to  plan. 
Include  an  RGC  in  the  TEMP  beginning  at  MS  B.  State  RGCs  in  a  series  of  intermediate  goals  that 
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can  be  tracked  through  fully  integrated,  system-level  T&E  events  until  the  reliability  threshold  is 
achieved.  If  a  single  curve  is  not  adequate  to  describe  overall  system  reliability,  curves  will  be 
provided  for  critical  subsystems  with  rationale  for  their  selection. 

•  PMs  and  OTAs  shall  assess  the  reliability  growth  required  for  the  system  to  achieve  its  reliability 
threshold  during  lOT&E  and  report  (progress  to  plan)  the  results  of  that  assessment  to  the  MDA 
at  MS  C  (Reference  (o)). 

17.3  Aspects  Of  LOG  T&E 

17.3.1  Process  for  Conducting  Logistics  Support  T&E 

LOG  T&E  measures  the  supportability  of  a  developing  system  throughout  the  acquisition  process  to 
identify  supportability  deficiencies  and  potential  corrections/improvements  as  test  data  become 
available  and  to  assess  the  operational  suitability  of  the  planned  support  system.  It  also  evaluates  the 
system's  ability  to  achieve  planned  readiness  objectives  for  the  system/equipment  being  developed. 
Specific  LOG  T&E  tasks  include  the  following: 

•  Determining  improvements  in  supportability  and  supportability-related  design  parameters 
needed  for  the  system  to  meet  established  goals  and  thresholds. 

•  Identifying  areas  in  which  established  goals  and  thresholds  have  not  been  demonstrated  within 
acceptable  confidence  levels. 

•  Developing  corrections  for  identified  supportability  problems  such  as  modifications  to 
hardware,  software,  support  plans,  logistics  support  resources,  or  operational  tactics. 

•  Projecting  changes  in  costs,  readiness,  and  logistics  support  resources  due  to  implementation  of 
corrections. 

•  Analyzing  supportability  data  from  the  deployed  system  to  verify  achievement  of  the 
established  goals  and  thresholds,  and  for  situations  in  which  operational  results  deviate  from 
projections,  determination  of  the  causes  and  corrective  actions. 

LOG  T&E  may  consist  of  a  series  of  logistics  demonstrations  and  assessments  that  are  usually  conducted 
as  part  of  system  performance  tests  but  may  require  dedicated  T&E.  Logistics  support  systems  may  also 
be  evaluated  as  part  of  an  R&M  test  event  such  as  a  maintenance  demonstration.  Special  end-item 
equipment  tests  are  rarely  conducted  solely  for  logistics  parameter  evaluation. 

17.3.2  Testing  for  Supportability 

Supportability  testing  is  conducted  to  ensure  that  requirements  for  the  12  IPS  elements  are  verified  (DT 
supportability  testing)  and  validated  (OT  suitability  testing,  supportability  T&E).  Determining  these 
requirements  is  not  always  a  straightforward  process  and  therefore  should  be  started  as  early  in  the 
process  as  possible.  At  the  establishment  of  the  T&E  WIPT,  the  PSM  should  appoint  a  logistician 
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representative  to  that  team  specifically  to  develop  supportability  T&E  objectives  and  monitor  their 
testing.  The  T&E  WIPT  should  be  established  as  early  as  possible  during  Material  Solutions  Analysis. 

One  of  the  first  tasks  the  LOG  T&E  representative  should  undertake  is  to  contact  the  OTA  suitability 
representative  to  lay  groundwork  for  future  integrated  testing.  Although  involvement  by  OTA 
representatives  will  be  limited  at  first,  it  is  never  too  early  to  bring  them  into  the  process. 

The  second  and  more  formidable  task  for  the  logistician  is  to  identify  all  of  the  supportability 
requirements  to  be  tested.  Some  of  these  requirements  are  listed  verbatim  in  the  requirements 
documents  (ICD/CDD/CPD)  as  KPPs  or  KSAs.  Examples  may  include  logistics  footprint,  supply  availability, 
mean  logistics  delay  time,  etc.  For  these  requirements,  all  that  the  LOG  T&E  representative  must  do  is 
read  the  documents  and  identify  the  logistics  requirements  as  stated. 

It  is  far  more  difficult  to  determine  implied  or  derived  supportability  requirements.  For  example,  if  there 
is  an  operational  availability  requirement  for  the  weapon  system  stated  in  the  CDD,  by  definition  it 
follows  that  supply  availability  and  component  delivery  times  will  have  to  be  at  a  certain  level.  Supply 
availability  and  component  delivery  times  are  subsets  of  mean  logistics  delay  time  (MLDT),  which  is  part 
of  the  downtime  in  the  operational  availability  equation.  Most  likely,  several  other  IPS  elements  will  be 
impacted  as  well.  (See  the  explanation  of  operational  availability  in  section  17.3.3.2  of  this  guide  for  the 
explanation  of  the  equation.)  Design  engineers  should  have  reliability  block  diagrams  that  derive  and 
allocate  the  systems  operational  availability  parameters  to  the  component  level  and  define  the 
equipment's  MLDT,  mean  time  to  repair  (MTTR),  and  appropriate  design  parameters,  which  will  be 
validated  during  testing. 

Other  types  of  derived  or  implied  requirements  are  those  generated  in  preparation  for  meeting 
suitability  COIs  in  OT.  For  example,  there  may  not  be  a  requirement  in  the  CDD  to  have  accurate,  logical, 
well-written  technical  manuals,  but  having  these  manuals  is  an  obvious  implied  requirement  for 
maintenance  personnel  to  do  their  job.  Because  of  this  implied  requirement,  it  is  a  certainty  that  the 
OTA  will  evaluate  technical  publications;  therefore,  it  behooves  the  program  office  to  evaluate  the 
publications  in  DT  to  avoid  unpleasant  surprises  in  OT.  Similarly,  requirements  for  all  12  IPS  elements 
must  be  developed  by  the  LOG  T&E  representative.  (See  paragraph  17.2.1  for  explanation  of  logistics 
suitability  and  COIs.) 

Both  derived  and  implied  requirements  should  be  worked  in  cooperation  with  the  OTA,  either  through 
joint  development  of  test  plans/assessments  or  by  comparing  independent  assessments  of  the 
requirements  documents  to  ensure  that  opportunities  for  integrated  testing  are  not  overlooked.  At  a 
minimum,  LOG  T&E  representatives  should  forward  the  requirements  (those  requirements  stated  in  the 
ICD/CDD/CPD  and  those  derived  from  other  requirements  they  have  identified)  to  the  OTA  suitability 
representative  for  concurrence  so  that  there  are  no  surprises  during  OT.  Optimally,  the  OTAs  suitability 
representative  would  be  a  contributing  member  of  the  T&E  WIPT  from  its  initial  implementation. 

A  third  responsibility  of  the  LOG  T&E  representative  is  to  ensure  that  logistics  products  are  available  in 
time  to  test.  It  is  critical  that  all  logistics  requirements  be  tested  during  DT.  Flistorically,  assets  for  testing 
logistics  requirements  are  delivered  too  late  in  the  acquisition  process  (i.e.,  too  close  to  lOT&E)  for  LOG 
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T&E  to  make  a  significant  difference  in  lOT&E.  If  there  are  serious  threats  to  long-term  suitability,  often 
logistics  requirements  testing  is  deferred  to  FOT&E,  which  is  of  limited  immediate  use  at  the  systems 
fielding.  The  LOG  T&E  representative  must  work  with  the  PSM  and  the  PM  to  accelerate  test  asset 
delivery  dates  so  that  any  needed  design  changes  can  be  implemented  prior  to  lOT&E. 

The  LOG  T&E  representative,  in  conjunction  with  the  PM,  PSM,  and  lead  for  T&E,  must  ensure  that  test 
personnel  are  trained  and  available,  that  there  is  enough  flexibility  in  the  test  facility  scheduling  to 
permit  normal  delays,  and  that  the  test  support  package  is  on-site  and  delivered  on  time. 

Once  all  requirements  are  identified,  the  LOG  T&E  representative  must  write  the  test  plans  that  will  be 
used  in  DT  so  that  the  requirements  will  be  tested  and  verified.  This  activity  should  be  done  via 
coordination  with  the  test  lead.  The  OTA  will  write  the  test  plans  for  OT.  Again,  the  optimal  solution  is 
one  of  cooperation  between  the  LOG  T&E  representative  and  suitability  test  personnel  working  in  an 
environment  of  integrated  testing.  Because  test  plan  preparation  procedures  vary  throughout  the 
Services,  additional  information  is  provided  in  each  agency's  T&E  policies  and  regulations. 

Often,  test  plans  for  supportability  T&E  during  DT  involve  the  LOG  T&E  representative  interviewing 
maintenance  technicians  over  the  shoulder  or  requesting  maintainers  to  complete  survey  forms  after 
each  maintenance  action.  These  forms  assist  in  determining  whether  the  IPS  element(s)  pertaining  to 
the  subsystem/component/part  under  consideration  meets  the  requirements.  These  surveys  are 
gathered  throughout  the  DT  period  and  are  compiled,  usually  in  databases,  so  that  at  any  time,  test 
personnel  can  request  the  supportability  posture  of  a  given  part  number/work  unit  code/logistics 
control  number/etc.  The  composite  score  of  multiple  technicians  working  with  the  same  component 
over  an  extended  time  is  then  used  as  an  indicator  to  determine  whether  further  research  is  required  as 
to  why  supportability  is  substandard  for  that  component.  The  LOG  T&E  representative  would  then 
research  quantifiable  maintenance  data  to  determine  causes  for  supportability  issues. 

Alternately,  the  supportability  evaluation  could  be  performed  via  logistics  demonstrations  (LOG  Demos). 
Like  supportability  T&E,  LOG  Demos  are  conducted  to  confirm  that  support  resources  and  tasks  that  are 
developed  to  sustain  the  system  will  function  as  intended.  Both  supportability  T&E  and  LOG  Demo  data 
are  supplemented  by  supportability-related  data  obtained  during  other  types  of  testing. 

The  LOG  Demo  normally  consists  of  the  following: 

•  Disassembly/Reassembly.  All  operator  tasks  and  specified  field-level  maintenance  tasks 
contained  in  the  technical  manuals  are  performed  by  personnel  representative  of  those  skill 
levels  typically  found  in  field  activities.  This  portion  of  the  LOG  Demo  also  serves  to  review  the 
technical  and  other  support  manuals  for  accuracy  and  readability. 

•  Maintainability  and  prognostics/embedded  diagnostics  demonstration.  Maintenance  and 
troubleshooting  tasks  as  the  result  of  operations  or  fault  simulation  or  insertion  are  to  be 
accomplished  by  typical  user  personnel  to  meet  the  intent  of  MILHDBK470A  (Reference  (bd)), 
DoDD  4151.18  (Reference  (be)),  and  DoDI  4151.22  (Reference  (bf)).  Complex  systems  may 
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require  additional  demonstrations  prior  to  fielding  of  test  program  sets  or  other  support 
capabilities  that  are  not  available  for  the  initial  fielding. 

Supportability  T&E  and  LOG  Demos  complement  each  other  and,  in  a  perfect  world,  both  would  be 
performed  during  evaluation  of  the  support  system  and  products.  With  LOG  Demos,  specific,  high- 
interest  components/systems  can  be  targeted  for  evaluation,  whereas  supportability  T&E  provides  long¬ 
term,  multi-maintenance  technician  perspectives  on  other  areas  of  supportability.  Both  are  useful  and 
should  be  utilized  in  verifying/evaluating  the  12  IPS  elements. 

17.3.3  RAM 

RAM  requirements  should  be  based  on  operational  requirements  and  LCC  considerations;  stated  in 
quantifiable,  operational  terms;  measurable  during  DT&E  and  OT&E;  and  defined  for  all  elements  of  the 
system,  including  support  and  training  equipment.  It  is  important  to  understand  that  RAM  is  a  function 
of  design  not  logistics.  RAM  is  designed  into  the  system  based  on  capability  needs  and  user 
requirements  and  is  augmented  by  supportability  measures  (IPS  elements).  The  tested  equipments  R&M 
design  features  have  a  great  influence  over  its  supportability.  For  example,  if  the  support  system  (12  IPS 
elements)  is  developed  concurrently  with  equipment  design,  and  some  aspect  of  the  system  fails  to 
meet  the  predicted  level  of  performance  (reliability  or  maintainability),  the  equipment  must  be 
redesigned  or  modifications  must  be  made  to  the  affected  product  support  element(s)  to  compensate 
for  the  failed  design  feature.  The  risk  of  not  mitigating  the  R&M  design  requirements  will  result  in 
additional  maintenance  to  keep  the  system  operational  and  lead  to  unexpected  failures  that  will  draw 
on  the  supply  chain  at  an  unplanned  rate  of  demand. 

17.3.3.1  Reliability 

Reliability  is  the  ability  of  a  system  and  its  parts  to  perform  its  mission  without  failure,  degradation,  or 
demand  on  the  support  system  under  a  prescribed  set  of  conditions.  A  common  MOE  is  mean  time 
between  failures  (MTBF)  or  mean  time  between  operational  mission  failures  (MTBOMF).  MTBF  is 
defined  as  a  measure  of,  for  a  particular  interval,  the  total  functional  life  of  a  population  of  an  item 
divided  by  the  total  number  of  failures  (requiring  corrective  maintenance  actions)  within  the  population. 
MTBOMF  is  the  measurement  for  mission-essential  equipment  to  operate  without  failure  during  a 
defined  mission  timeline  or  event.  Another  related  measure  is  mean  time  between  maintenance 
(MTBM),  a  measure  of  reliability  that  represents  the  average  time  between  all  maintenance  actions, 
both  corrective  and  preventive. 

Reliability  considerations  include  the  following: 

•  Environmental  Stress  Screening  (ESS) .  A  test  or  series  of  tests  during  engineering  development 
specifically  designed  to  identify  weak  parts  or  manufacturing  defects.  Test  conditions  should 
simulate  failures  typical  of  early  field  service  rather  than  provide  an  operational  life  profile. 

RDT/RGT .  A  systematic  engineering  process  of  TAFT  in  which  equipment  is  tested  under  actual, 
simulated,  or  accelerated  environments.  It  is  an  iterative  methodology  intended  to  rapidly  and  steadily 
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improve  reliability.  Test  articles  are  usually  subjected  to  ESS  prior  to  beginning  RDT/RGT  to  eliminate 
those  with  manufacturing  defects.  Reliability  growth  and  its  projection  can  be  depicted  via  RGCs,  which 
are  required  to  be  part  of  the  SEP  early  in  the  program  life  cycle  and  updated  in  the  TEMP  as  the 
program  matures.  Figure  17-4  provides  an  example  of  an  RGC,  taken  from  MIL-HDBK-189C  (Reference 
(bg)).  This  RGC  was  developed  via  the  Planning  Model  Based  on  Projection  Methodology  (PM2)-Discrete, 
one  of  several  reliability  growth  planning  model  options. 


Figure  17-4  Example  RGC 

•  Reliability  Qualification  Test .  A  test  to  verify  that  threshold  reliability  requirements  have  been 
met,  before  items  are  committed  to  production.  A  statistical  test  plan  is  used  to  predefine 
criteria,  which  will  limit  Government  risk.  Test  conditions  must  be  operationally  realistic. 

•  Production  Reliability  Acceptance  Test  (PRAT) .  A  test  intended  to  simulate  in-service  use  of  the 
delivered  item  or  production  lot.  Because  it  must  provide  a  basis  for  determining  contractual 
compliance  and  because  it  applies  to  the  items  actually  delivered  to  operational  forces,  PRAT 
must  be  independent  of  the  supplier  if  at  all  possible.  PRAT  may  require  expensive  test  facilities, 
so  100  percent  sampling  is  not  recommended. 

DoD  is  attempting  to  enhance  reliability  in  the  acquisition  process.  PMs  and  PMOs  are  required  to 
institutionalize  reliability  planning  methods  and  reporting  requirements  to  monitor  reliability  growth  in 
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accordance  with  Reference  (o) .  RGCs  will  be  stated  in  a  series  of  intermediate  goals  and  tracked 
through  fully  integrated,  system-level  T&E  events  until  the  reliability  threshold  is  achieved.  If  a  single 
curve  is  not  adequate  to  describe  overall  system  reliability,  curves  will  be  provided  for  critical 
subsystems  with  rationale  for  their  selection.  PMs  and  OTAs  shall  assess  the  reliability  growth  required 
for  the  system  to  achieve  its  reliability  threshold  during  lOT&E  and  report  the  results  of  that  assessment 
to  the  MDA  at  MS  C.  PMs  will  report  the  status  of  reliability  objectives  and/or  thresholds  as  part  of  the 
formal  design  review  process,  during  program  support  reviews,  and  during  SETRs.  RGCs  must  be 
employed  to  report  reliability  growth  status  at  DAES  reviews. 

17.3.3.2  Availability 

Availability  is  a  measure  of  the  degree  to  which  an  item  is  in  an  operable  state  and  can  be  committed  at 
the  start  of  a  mission  when  the  mission  is  called  for  at  an  unknown  (random)  point  in  time.  The  four 
availability  measurements  are  as  follows: 

•  Achieved  Availability  { A  a  ) .  Measured  for  the  early  prototype  and  EDM  systems  developed  by 
the  system  contractor  when  the  system  is  not  operating  in  its  normal  environment.  Availability 
of  a  system  with  respect  to  operating  time  and  both  corrective  and  preventive  maintenance.  It 
ignores  MLDT  and  may  be  calculated  as  MTBM  divided  by  the  sum  of  MTBM  and  mean 
maintenance  time  (MMT);  that  is: 

Aa  =  MTBM/(MTBM  +  MMT  ). 

•  Inherent  Availability  { A , ) .  Availability  of  a  system  with  respect  only  to  operating  time  and 
corrective  maintenance.  It  may  be  measured  during  DT&E  or  when  testing  in  the  contractor's 
facility  under  controlled  conditions.  A,  ignores  standby  and  delay  times  associated  with 
preventive  maintenance  as  well  as  MLDT  and  may  be  calculated  as  the  ratio  of  MTBF  divided  by 
the  sum  of  MTBF  and  MTTR;  that  is: 

A|  =  MTBF/(MTBF  +  MTTR). 

•  Operational  Availability  { Ao ) .  Measured  for  mature  systems  that  have  been  deployed  in 
realistic  operational  environments,  Aq  is  the  degree  to  which  a  piece  of  equipment  works 
properly  when  it  is  required  for  a  mission.  The  degree  (expressed  as  a  decimal  between  0  and  1, 
or  the  percentage  equivalent)  to  which  one  can  expect  a  piece  of  equipment  or  weapon  system 
to  work  properly  when  it  is  required;  that  is,  the  percent  of  time  the  equipment  or  weapon 
system  is  available  for  use.  Aq  represents  system  uptime  and  considers  the  effect  of  reliability, 
maintainability,  and  MLDT.  Aq  may  be  calculated  by  dividing  MTBM  by  the  sum  of  MTBM,  MMT, 
and  MLDT;  that  is: 

Ao  =  MTBM/(MTBM  +  MMT  +  MLDT).  Aq  is  the  quantitative  link  between  readiness  objectives 
and  supportability. 

•  Materiel  Availability  (A  m  )  ■  A  measure  of  the  percentage  of  the  total  inventory  of  a  system 
operationally  capable  (ready  for  tasking)  of  performing  an  assigned  mission  at  a  given  time, 
based  on  materiel  condition.  This  can  be  expressed  mathematically  as  the  number  of 
operational  end  items/total  population.  Materiel  availability  addresses  the  total  population  of 
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end  items  planned  for  operational  use,  including  those  temporarily  in  a  nonoperational  status 
once  placed  into  service  (such  as  for  depot-level  maintenance).  The  total  life-cycle  timeframe, 
from  placement  into  operational  service  through  the  planned  end  of  service  life,  must  be 
included.  This  is  often  referred  to  as  equipment  readiness.  Development  of  the  materiel 
availability  metric  is  a  PM  responsibility. 

17.3.3.3  Maintainability 

Maintainability  is  the  ability  of  an  item  to  be  retained  in  or  restored  to  a  specified  condition  when 
maintenance  is  performed  by  personnel  having  specific  skill  levels,  using  prescribed  procedures  and 
resources,  at  each  prescribed  level  of  maintenance  and  repair. 

A  common  maintainability  MOS  is  MTTR.  MTTR  is  the  total  elapsed  time  (clock  hours)  for  corrective 
maintenance  divided  by  the  total  number  of  corrective  maintenance  actions  during  a  given  period.  It  is  a 
basic  technical  measure  of  maintainability,  and  contractually,  time  and  repair  must  be  carefully  defined 
for  contractual  compliance  purposes. 

Maintainability  design  factors  and  test/demonstration  requirements  used  to  evaluate  maintainability 
characteristics  must  be  based  on  program  objectives  and  thresholds.  Areas  for  evaluation  might  include 
the  following: 

•  Accessibility  .  Assess  how  easily  the  item  can  be  repaired  or  adjusted. 

•  Visibility  .  Assess  the  ability/need  to  see  the  item  being  repaired. 

•  Testability  .  Assess  ability  to  detect  and  isolate  system  faults  to  the  faulty  replaceable  assembly 
level. 

•  Complexity  .  Assess  the  impact  of  the  number,  location,  and  characteristic  (standard  or  special 
purpose)  on  system  maintenance. 

•  Interchangeability  .  Assess  the  level  of  difficulty  encountered  when  failed  or  malfunctioning 
parts  are  removed  or  replaced  with  an  identical  part  not  requiring  recalibration. 

A  true  assessment  of  system  maintainability  generally  must  be  developed  at  the  system  level  under 
operating  conditions  and  using  production  configuration  hardware.  Therefore,  a  maintainability 
demonstration  should  be  conducted  prior  to  the  FRP  decision.  The  risk  of  not  mitigating  the  R&M  design 
requirements  will  result  in  additional  maintenance  to  keep  the  system  operational  and  lead  to 
unexpected  failures  that  will  draw  on  the  supply  chain  at  an  unplanned  rate  of  demand. 

17.3.3.4  Operational  R&M  Value 

The  operational  R&M  value  is  any  measure  of  reliability  or  maintainability  that  includes  the  combined 
effects  of  item  design,  quality,  installation,  environment,  operation,  maintenance,  and  repair. 
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RAM  program  objectives  are  defined  as  system  parameters  derived  from  capability-driven  KPPs  early  in 
the  design  and  development  process.  They  will  be  used  as  evaluation  criteria  throughout  the  design, 
development,  and  production  processes.  These  objectives  should  be  translated  into  quantifiable 
contractual  terms  and  ultimately  will  be  allocated  via  the  SE  process  through  the  system  design 
hierarchy  to  specific  components. 

A  more  detailed  description  about  how  this  allocation  flow-down  process  affects  RAM,  among  other 
topics,  can  be  found  in  the  DoD  Guide  for  Achieving  Reliability,  Availability,  and  Maintainability 
(Reference  (bh)).  The  guide  notes  that  RAM  and  its  measurement  will  vary  over  the  life  cycle,  stating: 

Many  factors  are  important  to  RAM:  system  design;  manufacturing  quality;  the  environment  in  which 
the  system  is  transported,  handled,  stored,  and  operated;  the  design  and  development  of  the  support 
system;  the  level  of  training  and  skills  of  the  people  operating  and  maintaining  the  system;  the 
availability  of  materiel  required  to  repair  the  system;  and  the  diagnostic  aids  and  tools  (instrumentation) 
available  to  them.  All  these  factors  must  be  understood  to  achieve  a  system  with  a  desired  level  of  RAM. 
During  pre-systems  acquisition,  the  most  important  activity  is  to  understand  the  user's  needs  and 
constraints.  During  system  development,  the  most  important  RAM  activity  is  to  identify  potential  failure 
mechanisms  and  to  make  design  changes  to  remove  them.  During  production,  the  most  important  RAM 
activity  is  to  ensure  quality  in  manufacturing  so  that  the  inherent  RAM  qualities  of  the  design  are  not 
degraded. 

17.3.4  T&E  of  Training  Materials,  Equipment  and  Facilities 

The  development  and  subsequent  T&E  of  trainers  and  training  equipment  is  an  activity  that  closely 
parallels  the  activity  associated  with  system  acquisition  events.  Design  engineering,  system  integration, 
support  system  identification,  and  development,  manufacturing,  and  production  processes  are  all 
applicable  to  trainers  and  training  equipment.  Occasionally,  however,  milestone  schedules  may  require 
delivery  of  the  trainers  and  training  equipment  before  or  at  the  same  time  as  the  major  system.  Trainers 
and  training  equipment  must  be  identical  representations  of  the  system  they  support;  therefore,  their 
development  must  follow  system  development.  Expeditious  completion  of  all  engineering  and  logistics 
tasks  is  necessary  in  order  to  meet  delivery  schedules  and  to  complete  T&E  activities. 

17.3.5  Data  Collection  and  Analysis 

The  logistics  support  manager  should  coordinate  with  the  testers  to  ensure  that  the  methods  used  for 
collection,  storage,  and  extraction  of  LOG  T&E  data  are  compatible  with  those  used  in  testing  the 
materiel  system.  As  with  any  testing,  the  test  planning  must  ensure  that  all  required  data  are  identified; 
that  the  data  are  sufficient  to  evaluate  a  systems  readiness  and  supportability;  and  that  plans  are  made 
for  a  data  management  system  that  is  capable  of  the  data  classification,  storage,  retrieval,  and  reduction 
necessary  for  statistical  analysis.  Large  statistical  sample  sizes  may  require  a  common  database  that 
integrates  contractor,  DT&E,  integrated  testing,  and  OT&E  data  so  required  performance  parameters 
can  be  demonstrated. 
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17.3.6  Use  of  LOG  T&E  Results 

The  emphasis  on  the  use  of  the  results  of  testing  changes  as  the  program  moves  from  initiation  to  post¬ 
deployment.  During  early  phases  of  a  program,  the  evaluation  results  are  used  primarily  to  verify 
analysis  and  develop  future  projections.  As  the  program  moves  into  advanced  engineering  development 
and  hardware  becomes  available,  the  evaluation  addresses  design,  particularly  the  R&M  aspects; 
trainers  and  training  programs;  support  equipment  adequacy;  personnel  skills  and  availability;  adequacy 
of  supply  support;  computer  resources;  and  technical  publications. 

The  PSM  must  make  the  PM  aware  of  the  impact  on  the  program  of  logistical  shortcomings  that  are 
identified  during  the  T&E  process.  The  PM  in  turn  must  ensure  that  the  solutions  to  any  shortcomings 
are  identified  and  reflected  in  the  revised  specifications  and  that  the  revised  test  requirements  are 
included  in  the  updated  TEMP  as  the  program  proceeds  through  the  various  acquisition  stages. 

17.4  Limitations  to  LOG  T&E 

Concurrent  testing  or  tests  that  have  accelerated  schedules  frequently  do  not  have  sufficient  test 
articles,  equipment,  or  hardware  to  achieve  statistical  confidence  in  the  testing  conducted.  An 
acquisition  strategy  may  stipulate  that  support  resources  such  as  operator  and  maintenance  manuals, 
tools,  support  equipment,  training  devices,  etc.,  for  major  weapon  system  components  would  not  be 
procured  before  the  weapons  system/component  hardware  and  software  design  stabilizes.  This 
shortage  of  equipment  is  often  the  reason  that  shelf-life  and  service-life  testing  is  incomplete,  leaving 
the  logistics  support  evaluator  with  insufficient  data  to  predict  future  performance  of  the  test  item. 
Some  evaluations  must  measure  performance  against  a  point  on  the  parameters  growth  curve.  The 
logistics  support  testing  will  continue  post-production  to  obtain  required  sample  sizes  for  verifying 
performance  criteria.  Some  aspects  of  the  logistics  support  may  not  be  available  for  lOT&E  and  become 
testing  limitations  requiring  waivers.  The  PMO  must  develop  enough  logistics  support  to  ensure  that  the 
user  can  maintain  the  system  during  lOT&E  without  requiring  system  contractor  involvement  (legislated 
constraints).  Any  logistics  support  limitations  upon  lOT&E  will  need  to  be  evaluated  during  FOT&E. 

17.5  Summary 

T  &E  is  a  key  tool  for  measuring  the  ability  of  the  planned  support  system  to  fulfill  the  materiel  systems 
readiness  and  supportability  objectives.  The  effectiveness  of  the  LOG  T&E  effort  is  based  upon  the 
completeness  and  timeliness  of  the  planning  effort.  A  comprehensive  R&M  program  using  an 
appropriate  reliability  growth  strategy  to  improve  R&M  performance  will  be  executed  by  all  programs. 

The  LOG  T&E  requirements  must  be  an  integral  part  of  the  TEMP  to  ensure  budgeting  and  scheduling  of 
required  test  resources.  Data  requirements  must  be  completely  identified,  with  adequate  plans  made 
for  collection,  storage,  retrieval,  and  reduction  of  test  data.  At  the  FRPDR,  decision  makers  can  expect 
that  some  logistics  support  performance  parameters  may  not  have  finished  testing  because  of  the  large 
sample  sizes  required  for  high  confidence  levels  in  statistical  analysis. 
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Test  &  Evaluation  Management  Guide 
Chapter  18  -  M&S  Support  to  T&E 

18.1  Introduction 

This  chapter  discusses  the  applications  of  M&S  in  T&E.  DoD  policies  encourage  the  early  use  of  M&S  as  a 
key  tool  throughout  the  acquisition  life  cycle.  M&S,  when  properly  employed,  provides  the  T&E 
community  with  valuable  information  that  can  increase  confidence  levels,  decrease  field  test  time  and 
costs,  and  provide  data  for  pre-test  prediction  and  post-test  validation.  A  symbiotic  relationship  exists 
between  M&S  and  T&E.  All  DoD  M&S  requires  VV&A  for  each  use  in  the  acquisition  process,  and  T&E  is 
needed  to  validate  most  models.  M&S  must  be  well  understood  to  ensure  a  reasonable  return  on 
investment  for  its  use.  This  chapter  discusses  using  M&S  to  increase  the  efficiency  of  the  T&E  process, 
reduce  time  and  cost,  provide  otherwise  unattainable  and  immeasurable  data,  and  provide  more  timely 
and  valid  results. 

18.2  Types  of  Models  and  Simulations 

The  term  modeling  and  simulation  is  often  associated  with  huge  digital  computer  simulations,  but  it  also 
includes  manual  and  man-in-the-loop  war  games,  HWIL  simulations,  test  beds,  hybrid  laboratory 
simulators,  and  prototypes. 

A  mathematical  model  is  an  abstract  representation  of  a  system  that  provides  a  means  of  developing 
quantitative  performance  requirements  from  which  candidate  designs  can  be  developed.  Static  models 
are  those  that  depict  conditions  of  state,  whereas  dynamic  models  depict  conditions  that  vary  with  time, 
such  as  the  action  of  an  autopilot  in  controlling  an  aircraft.  Simple  dynamic  models  can  be  solved 
analytically,  and  the  results  represented  graphically. 

As  illustrated  in  Figure  18-1,  there  is  a  broad  spectrum  of  simulation  types  ranging  from 
theater/campaign  simulations  to  detailed  engineering  simulations.  A  program  will  likely  use  a  suite  of 
models  and  simulations.  The  engineering-level  models  assess  MOPs  along  with  design,  cost, 
producibility,  and  supportability  information  for  components,  subsystems,  and  systems.  The  military 
utility  of  systems  can  be  evaluated  within  engagement  and  mission/battle-level  models.  At  the  highest 
level,  theater/campaign  models  can  be  used  to  simulate  the  outcomes  of  major  conflicts.  Clearly,  there 
is  no  single  model  or  simulation  to  suit  all  of  a  programs  needs.  Each  model  or  simulation  has  a  specific 
purpose  for  which  it  was  intended  and  the  tester  should  carefully  evaluate  its  utility  for  T&E  purposes. 
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Figure  18-1  The  Simulation  Spectrum 

One  way  of  looking  at  the  role  of  simulations  used  in  T&E  is  by  categorizing  the  simulations  into  a  three- 
part  continuum,  which  is  described  below. 

Constructive  Simulations  .  Constructive  simulations  are  strictly  mathematical  representations  of 
systems  and  do  not  employ  any  actual  hardware.  They  involve  simulated  people  operating  simulated 
systems.  Real  people  stimulate  (provide  inputs  to)  constructive  simulations  but  are  not  involved  in 
determining  the  outcomes.  Constructive  simulations  may,  however,  incorporate  some  of  the  actual 
software  that  might  be  used  in  a  system.  Early  in  a  systems  life  cycle,  computer  simulations  may  provide 
the  most  system  evaluation  information.  In  many  cases,  computer  simulations  can  be  readily  developed 
as  modifications  of  existing  simulations  for  similar  systems. 


Virtual  Simulations  .  Virtual  simulations  involve  real  people  operating  simulated  systems.  A  system  test 
bed  usually  differs  from  a  computer  simulation  as  it  contains  some,  but  not  necessarily  all,  of  the  actual 
hardware  that  will  be  a  part  of  the  system.  Other  elements  of  the  system  are  either  not  incorporated  or, 
if  they  are  incorporated,  are  in  the  form  of  computer  simulations.  The  system  operating  environment 
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(including  threat)  may  either  be  physically  simulated,  as  in  the  case  of  a  flying  test  bed,  or  computer 
simulated,  as  in  the  case  of  a  laboratory  test  bed.  Aircraft  cockpit  simulators  used  to  evaluate  pilot 
performance  are  good  examples  of  system  test  beds.  As  development  of  a  system  progresses,  more 
subsystems  become  available  in  hardware  form.  These  subsystems  can  be  incorporated  into  system  test 
beds  that  typically  provide  a  great  deal  of  the  system  evaluation  information  used  during  the  middle 
part  of  a  systems  development  cycle. 

Another  type  of  virtual  simulation  used  in  T&E  is  the  system  prototype.  Unlike  the  system  test  bed,  all 
subsystems  are  physically  incorporated  in  a  system  prototype.  The  system  prototype  may  closely 
represent  the  final  system  configuration,  depending  on  the  state  of  development  of  the  various 
subsystems  that  compose  it.  Preproduction  prototype  missiles  and  aircraft  used  in  OT  by  the  Services 
are  examples  of  this  class  of  simulation.  As  system  development  proceeds,  eventually  all  subsystems  will 
become  available  for  incorporation  in  one  or  more  system  prototypes.  HWIL  simulators  or  full-up  man- 
in-the-loop  system  simulators  may  provide  the  foundation  for  continuous  system  testing  and 
improvement.  These  simulators  can  provide  the  basis  for  transitioning  hardware  and  software  into 
operationally  realistic  training  devices  with  mission  rehearsal  capabilities.  OT  of  these  prototypes 
frequently  provides  much  of  the  system  evaluation  information  needed  for  a  decision  on  full-scale 
production  and  deployment. 

Live  Simulations  .  Live  simulations  involve  real  people  operating  real  systems.  Some  say  that  everything 
except  global  combat  is  a  simulation,  even  limited  regional  engagements.  Live  exercises  in  which  troops 
use  equipment  under  actual  environmental  conditions  approach  real  life  in  combat  while  conducting 
peacetime  operations.  Training  exercises  and  other  live  simulations  provide  a  testing  ground  with  real 
data  on  actual  hardware,  software,  and  human  performance  when  subjected  to  stressful  conditions. 
These  data  can  be  used  to  validate  the  models  and  simulations  used  in  an  acquisition  program. 

The  complexity  of  testing  in  a  joint  environment  will  require  testing  in  a  distributed  environment  that 
combines  live,  virtual,  and  constructive  models.  Hardware,  software,  databases,  and  networks  are 
integrated  into  a  system  and  tested  to  make  sure  they  can  operate  as  intended. 

M&S  accuracy  is  dependent  on  the  technical  level  of  confidence  or  credibility  of  the  simulation.  To 
ensure  that  confidence  in  a  particular  use  of  M&S  for  a  given  purpose  is  justified,  a  rigorous  process 
must  be  followed,  such  that  (1)  Modeling  assumptions  are  accurate  and  well  documented;  (2)  Results 
produced  by  the  M&S  are  stable,  consistent,  and  repeatable;  and  (3)  The  correlation  between  the  M&S 
behavior  and  real  world  behavior  is  clearly  understood. 

Model  credibility  is  established  via  a  VV&A  set  of  increasingly  rigorous  criteria.  DoDI  5000.61  (Reference 
(bi))  defined  VV&A  as  follows: 

•  Verification  .  The  process  of  determining  that  a  model  or  simulation  implementation  and  its 
associated  data  accurately  represent  the  developer's  conceptual  description  and  specifications. 
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•  Validation  .  The  process  of  determining  the  degree  to  which  a  model  or  simulation  and  its 
associated  data  are  an  accurate  representation  of  the  real  world  from  the  perspective  of  the 
intended  uses  of  the  model. 

•  Accreditation  .  The  official  certification  that  a  model  or  simulation  and  its  associated  data  are 
acceptable  for  use  for  a  specific  purpose. 

18.3  Scope  of  M&S 

Simulations  are  not  a  substitute  for  live  testing.  There  are  many  things  that  cannot  be  adequately 
simulated  by  computer  programs,  including  the  decision  process  and  the  proficiency  of  personnel  in  the 
performance  of  their  functions.  Therefore,  models  and  simulations  are  not  a  total  substitution  for 
physical  T&E  events.  Simulations,  manual  and  computer-designed,  can  complement  and  increase  the 
validity  of  live  T&E  events  by  proper  selection  and  application.  Figure  18-2  contrasts  the  test  criteria 
values  that  are  conducive  to  physical  testing  with  those  conducive  to  M&S.  Careful  selection  of  the 
simulation,  knowledge  of  its  application,  and  meticulous  selection  of  input  data  will  produce 
representative  and  valid  results. 
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Figure  18-2.  Criteria  Values  Conducive  to  Use  of  M&S 
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Conversely,  certain  highly  complex  simulations  can  be  more  effective  analytical  tools  than  limited  live 
testing,  generally  due  to  the  significance  of  the  external  constraints.  For  example,  nuclear  weapons 
effects  testing  can  be  done  only  in  simulation  because  of  international  treaties.  Testing  depleted 
uranium  ammunition  is  very  expensive  with  its  related  environmental  considerations.  Finally,  much 
confidence-building  iteration  of  multimillion-dollar  missile  defense  tests  can  be  executed  using  only 
simulated  events  in  conjunction  with  a  few  instances  of  real  firings. 

The  important  element  in  using  a  simulation  is  to  select  one  that  is  representative  and  either  addresses 
or  is  capable  of  being  modified  to  address  the  level  of  detail  (issues,  thresholds,  and  objectives)  under 
investigation. 

18.4  Support  to  Test  Design  and  Planning 

M&S  can  assist  in  the  T&E  planning  process,  may  reduce  the  cost  of  testing,  and  can  provide 
environments  and  sample  sizes  not  otherwise  achievable.  In  Figure  18-3,  areas  of  particular  application 
include  scenario  development  and  the  timing  of  test  events;  the  development  of  objectives,  essential 
elements  of  analysis,  and  MOEs;  the  identification  of  variables  for  control  and  measurement;  and  the 
development  of  data  collection,  instrumentation,  and  data  analysis  plans.  For  example,  using  simulation, 
the  test  designer  can  examine  system  sensitivities  to  changes  in  variables  to  determine  the  critical 
variables  and  their  ranges  of  values  to  be  tested.  The  tester  can  also  predict  the  effects  of  various 
assumptions  and  constraints  to  help  formulate  the  test  design.  For  example,  in  the  live-fire  vulnerability 
testing  of  the  F-22  wing  panel,  a  mechanical  deformation  model  was  used  extensively  in  pre-test 
predictions.  The  pre-test  analysis  was  used  to  design  the  experiment  that  assisted  in  shot-line  selection 
and  allowed  elimination  of  the  aft  boom  testing,  saving  test  schedule  and  about  $100,000.  Real  data  in 
post-test  analysis  verified  the  capability  to  predict  impulse  damage  within  acceptable  limits,  resulting  in 
greater  use  of  the  model  in  future  applications. 

During  pre-MS  A  and  the  TD  phase,  models  for  LFT&E  and  survivability  are  needed  to  help  establish 
design  factors  and  considerations  and  help  support  initial  vulnerability  assessment  reports.  These 
models  can  be  used  in  the  early  stages  of  development  and  if  properly  validated,  can  support  later 
stages  of  survivability  assessments.  Models  should  help  replicate  the  environment  during  test  to 
realistically  stress  the  system  under  test. 

Caution  must  be  exercised  when  planning  to  rely  on  simulations  to  obtain  test  data  as  they  tend  to  be 
expensive  to  develop  or  modify,  tend  to  be  difficult  to  integrate  with  data  from  other  sources,  and  often 
do  not  provide  the  level  of  realism  required  for  OTs.  Although  simulations  are  not  a  cure-all,  they  should 
be  explored  and  used  whenever  feasible  as  another  source  of  data  for  the  evaluator  to  consider  during 
the  test  evaluation.  Simulation  fidelity  is  improving  rapidly  with  enhancements  to  computer  technology. 

Computer  simulations  may  be  used  to  test  the  planning  for  an  exercise.  By  setting  up  and  running  the 
test  exercise  in  a  simulation,  the  timing  and  scenario  may  be  tested  and  validated.  Critical  events  may 
include  interaction  of  various  forces  that  test  the  MOEs  and  in  turn  the  test  objectives.  Further,  the 
simulation  may  be  used  to  verify  the  statistical  test  design  and  the  instrumentation,  data  collection,  and 
data  analysis  plans.  Essentially,  the  purpose  of  computer  simulation  in  pre-test  planning  is  to  preview 
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the  test  to  evaluate  ways  to  make  test  results  more  effective.  Pre-testing  attempts  to  optimize  test 
results  by  pointing  out  potential  trouble  spots.  It  constitutes  a  test  setup  analysis,  which  can  encompass 
a  multitude  of  areas.  The  model-test-model  process  is  an  integrated  approach  to  using  models  and 
simulations  in  support  of  pre-test  analysis  and  planning,  conducting  the  actual  test  and  collecting  data, 
and  performing  post -test  analysis  of  test  results  along  with  further  validation  of  the  models  using  the 
test  data. 

As  an  example  of  simulations  used  in  test  planning,  consider  a  model  that  portrays  aircraft  versus  air 
defenses.  The  model  can  be  used  to  replicate  typical  scenarios  and  provide  data  on  the  number  of 
engagements,  air  defense  systems  involved,  aircraft  target,  length  and  quality  of  the  engagement,  and  a 
rough  approximation  of  the  success  of  the  mission  (i.e.,  if  the  aircraft  made  it  to  the  target).  With  such 
data  available,  a  data  collection  plan  can  be  developed  to  specify,  in  more  detail,  when  and  where  data 
should  be  collected,  from  which  systems,  and  in  what  quantity.  The  results  of  this  analysis  impact 
heavily  on  long  lead-time  items  such  as  data  collection  devices  and  data  processing  systems.  The  more 
specificity  available  earlier,  the  fewer  the  number  of  surprises  that  will  occur  downstream  in  the 
development  process.  As  tactics  are  decided  upon  and  typical  flight  paths  are  generated  for  the 
scenario,  an  analysis  can  be  prepared  on  the  flight  paths  over  the  terrain  in  question,  and  a 
determination  can  be  made  regarding  whether  the  existing  instrumentation  can  track  the  numbers  of 
aircraft  involved  in  their  maneuvering  envelopes.  Alternative  site  arrangements  can  be  examined  and 
trade-offs  can  be  made  between  the  amount  of  equipment  to  be  purchased  and  the  types  of  profiles 
that  can  be  tracked  for  this  particular  test. 
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Figure  18-3  Illustrative  Uses  of  M&S  in  T&E 

Use  of  such  a  model  can  also  highlight  numerous  choices  available  to  the  threat  air  defense  system  in 
terms  of  opportunities  for  engagement  and  practical  applications  of  doctrine  to  the  specific  situations. 

18.5  Support  to  Test  Execution 

Simulations  can  be  useful  in  test  execution  and  dynamic  planning.  Funds  and  other  restrictions  may  limit 
the  number  of  times  a  test  may  be  repeated.  The  test  director  must  exercise  close  control  over  the 
conduct  of  the  test  to  ensure  that  the  specific  types  and  quantities  of  data  needed  to  meet  the  test 
objectives  are  being  gathered  and  to  ensure  adequate  safety.  The  test  director  must  be  able  to  make 
minor  modifications  to  the  test  plan  and  scenario  to  force  achievement  of  these  goals.  This  need  calls 
for  a  dynamic  (quick-look)  analysis  capability  and  a  dynamic  planning  capability.  Simulations  may 
contribute  to  this  dynamic  planning  capability.  For  example,  using  the  same  simulation(s)  as  used  in  pre¬ 
test  planning,  the  tester  could  input  data  gathered  during  the  first  day  of  the  exercise  to  determine  the 
adequacy  of  the  data  to  fulfill  the  test  objectives.  Using  this  data,  the  entire  test  could  be  simulated. 
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Projected  inadequacies  could  be  isolated,  and  the  test  plans  could  be  modified  to  minimize  the 
deficiencies. 

Simulations  may  also  be  used  to  support  test  control  and  to  ensure  safety.  For  example,  during  missile 
test  firings  at  White  Sands  Missile  Range,  New  Mexico,  aerodynamic  simulations  of  the  proposed  test 
were  run  on  a  computer  during  actual  firings  so  that  real-time  missile  position  data  could  be  compared 
continuously  to  the  simulated  missile  position  data.  If  any  significant  variations  occurred  and  if  the  range 
safety  officer  was  too  slow  (both  types  of  position  data  were  displayed  on  plotting  boards),  the 
computer  issued  a  destruct  command. 

Simulations  can  be  used  to  augment  tests  by  simulating  non-testable  events  and  scenarios.  Although  OT 
should  be  accomplished  in  as  realistic  an  operational  environment  as  possible,  pragmatically  some 
environments  are  impossible  to  simulate  for  safety  or  other  reasons;  for  example,  the  environment  of  a 
nuclear  battlefield,  to  include  the  effects  of  nuclear  bursts  on  friendly  and  enemy  elements.  Other 
examples  include  two-sided  live  firings  and  adequate  representation  of  other  forces  to  ascertain 
compatibility  and  interoperability  data.  Instrumentation,  data  collection,  and  data  reduction  of  large 
combined  armed  forces  (e.g..  Army  brigade  and  larger-sized  forces)  become  extremely  difficult  and 
costly.  Simulations  are  not  restricted  by  safety  factors  and  can  realistically  replicate  many  environments 
that  are  otherwise  unachievable  in  OT&E-nuclear  effects,  large  combined  forces,  ECM,  electronic 
counter-countermeasures,  and  many  engagements.  These  effects  can  be  simulated  repeatedly  with  no 
environmental  impacts,  costing  only  for  the  simulation  operations. 

Usually,  insufficient  units  are  available  to  simulate  the  organizational  relationships  and  interaction  of  the 
equipment  with  its  operational  environment,  particularly  during  the  early  OT&E  conducted  using 
prototype  or  pilot  production-type  equipment.  Simulations  are  not  constrained  by  these  limitations. 

Data  obtained  from  a  limited  test  can  be  plugged  into  a  simulation  that  is  capable  of  handling  many  of 
the  types  of  equipment  being  tested.  The  simulation  can  interface  the  units  with  other  elements  of  the 
blue  forces  and  operate  them  against  large  elements  of  the  red  forces  to  obtain  interactions. 

End-item  simulators  can  be  used  to  evaluate  design  characteristics  of  equipment  and  can  be  used  to 
augment  the  results  obtained  using  prototype  or  pilot  production-type  equipment  that  is  representative 
of  the  final  item.  The  simulator  may  be  used  to  expand  test  data  to  obtain  the  required  iterations  or  to 
indicate  that  the  human  interface  with  the  prototype  equipment  will  not  satisfy  the  design 
requirements. 

It  is  often  necessary  to  use  substitute  or  surrogate  equipment  in  testing;  for  example,  American 
equipment  is  used  to  represent  threat-force  equipment.  In  some  cases,  the  substitute  equipment  may 
have  greater  capabilities  than  the  real  equipment;  in  other  cases,  it  may  have  less.  Simulations  are 
capable  of  representing  the  real  characteristics  of  equipment  and,  therefore,  can  be  used  as  a  means  of 
modifying  raw  data  collected  during  the  test  to  reflect  real  characteristics. 

As  an  example,  if  the  substitute  equipment  is  an  anti-aircraft  gun  with  a  tracking  rate  of  30  degrees  per 
second  and  the  unavailable  threat  equipment  for  which  it  is  substituted  has  a  tracking  rate  of  45  degrees 
per  second,  the  computer  simulation  could  be  used  to  augment  the  collected,  measured  data  by 
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determining  how  many  rounds  could  have  been  fired  against  each  target,  or  whether  targets  that  were 
missed  because  the  tracking  rate  was  too  slow  could  have  been  engaged  by  the  actual  equipment. 
Consideration  of  other  differing  factors  simultaneously  could  have  a  positive  or  negative  synergistic 
effect  on  test  results  and  could  allow  evaluation  more  quickly  through  a  credible  simulation  designed 
with  the  flexibility  to  replicate  operational  variations. 

18.6  Support  to  Analysis  and  Test  Reporting 

M&S  may  be  used  in  post-test  analysis  to  extend  and  generalize  results  and  to  extrapolate  to  other 
conditions.  The  difficulty  of  instrumentation  for  controlling  large  exercises  and  for  collecting  and 
reducing  the  data  and  resource  costs  to  some  degree  limits  the  size  of  T&E.  This  difficulty  makes  the 
process  of  determining  the  suitability  of  equipment,  including  compatibility,  interoperability, 
organization,  etc.,  a  difficult  one.  To  a  large  degree,  the  limited  interactions,  interrelationships,  and 
compatibility  of  large  forces  may  be  supplemented  by  using  actual  data  collected  during  the  test  and 
applying  the  data  in  the  simulation. 

Simulations  may  be  used  to  extend  test  results,  save  considerable  energy  (fuel  and  manpower),  and  save 
money  by  reducing  the  need  to  repeat  data  points  to  improve  the  statistical  sample  or  to  determine 
overlooked  or  directly  unmeasured  parameters.  Sensitivity  analyses  can  be  run  using  simulations  to 
evaluate  the  robustness  of  the  design. 

In  analyzing  the  test  results,  data  may  be  compared  to  the  results  predicted  by  the  simulations  used 
early  in  the  planning  process.  Thus,  the  simulation  is  validated  by  the  actual  live  test  results,  and  the  test 
results  are  also  validated  by  the  simulation. 

18.7  Simulation  Integration 

Simulations  have  become  so  capable  and  widespread  in  their  application,  combined  with  the  ability  to 
network  dissimilar  and/or  geographically  separate  simulators  in  real  time  to  synergistically  simulate 
more  than  ever  before,  that  whole  new  applications  are  being  discovered.  Simulations  are  no  longer 
stove-piped  tools  in  distinct  areas  but  are  increasingly  crossing  disciplines  and  different  uses.  The  F-35 
Joint  Strike  Fighter  program  uses  more  than  200  different  models  and  simulations  throughout  its  many 
development  and  testing  activities,  in  both  Government  centers  and  contractor  facilities.  Increasingly, 
operational  issues  are  being  determined  through  operational  analysis  simulations,  which  will 
subsequently  impact  OT  much  later  in  a  program. 

OT  elements  of  the  T&E  community  should  be  involved  increasingly  earlier  in  every  program  to  ensure 
that  such  results  are  compatible  with  the  OT  objectives  and  requirements.  For  example,  the  F-35 
program  networked  eight  different  war  game  simulations  together  to  develop  the  initial  operational 
requirements  that  subsequently  were  documented  in  the  programs  requirements  document  for  the 
many  different  required  Service  applications  (i.e..  Navy,  Air  Force,  Marine  Corps,  and  British  Royal  Navy). 
Only  through  this  extensive  Virtual  Strike  Warfare  Environment  was  the  broad  and  concurrent  range  of 
operational  requirements  able  to  be  evaluated  and  established.  New  capabilities  were  achieved,  but  the 
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extensive  multi-simulation  network  demanded  new  techniques  for  establishing  the  technical  confidence 
in  the  results . 

18.8  Simulation  Planning 

With  M&S  becoming  increasingly  more  complex,  more  expensive,  and  more  extensive,  testers  must  be 
thorough  in  planning  their  use  of  M&S  and  subsequent  technical  confidence.  Testers,  including  the 
OTAs,  are  expected  to  be  involved  increasingly  earlier  if  the  subsequent  T&E  results  are  to  be  accepted. 
Simulation  support  plans  (SSPs)  are  program  documents  that  span  the  many  simulations,  their  purpose, 
and  their  expected  credibility.  The  Services  have  policies  requiring  early  M&S  planning.  Usually,  this 
process  starts  with  a  program  office-level  simulation  support  group  of  Service  M&S  experts  who  advise 
the  program  on  M&S  opportunities,  establish  the  VV&A  process,  and  document  these  procedures  in  the 
SSP,  which  extends  across  the  life  cycle  of  system  development,  testing,  and  employment.  It  is  vital  that 
the  SSP  be  fully  coordinated  with  the  TEMP.  Without  early  planning  of  what  each  model  or  simulation  is 
for,  its  expense,  and  its  impact  on  system  design  as  well  as  testing,  earlier  programs  have  ended  up  with 
a  volume  of  different  data  for  different  purposes,  which  could  not  be  analyzed,  was  subsequently  found 
not  to  be  credible,  or  delayed  testing  or  program  development. 

18.9  Summary 

M&S  in  T&E  can  be  used  for  concept  evaluation,  extrapolation,  isolation  of  design  effects,  efficiency, 
representation  of  complex  environments,  increasing  sample  size,  and  overcoming  inherent  limitations  in 
actual  testing.  The  use  of  M&S  can  validate  test  results,  increase  confidence  levels,  reduce  test  costs, 
and  provide  opportunities  to  shorten  the  overall  acquisition  cycle  by  providing  more  data  earlier  for  the 
decision  maker.  It  takes  appropriate  time  and  funding  to  bring  models  and  simulations  along  to  the 
point  that  they  are  useful  during  an  acquisition. 
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Test  &  Evaluation  Management  Guide 

Chapter  19  -  Electromagnetic  Environmental  Effects  (E3)  Testing  and  Spectrum 
Supportability  (SS) 

19.1  Introduction 

Historically,  failure  to  verify  equipment/platform  electromagnetic  compatibility  in  the  items  intended 
operational  electromagnetic  environment  has  caused  costly  program  delays  and  adversely  affected 
operational  safety,  suitability,  and  effectiveness.  E3  can  affect  the  performance  of  all  electrical  and 
electronic  systems,  subsystems,  and  equipment,  including  ordnance  containing  electrically  initiated 
devices.  Therefore,  any  system  that  has  electronic  components  must  be  reviewed  to  determine  the 
extent  of  E3  issues;  his  requirement  applies  to  more  than  just  communication  and  electronics  systems. 
Such  systems  must  be  mutually  compatible  in  their  intended  electromagnetic  environment  (EME) 
without  causing  or  suffering  unacceptable  mission  degradation  due  to  E3. 

DoDD  3222.3  (Reference  (bj))  provides  policy  and  responsibilities  for  the  management  and 
implementation  of  the  DoD  E3  Program  to  ensure  mutual  electromagnetic  compatibility  (EMC)  and 
effective  E3  control  among  ground,  air,  sea,  and  space-based  electronic  and  electrical  systems  and 
subsystems,  and  with  the  existing  natural  and  man-made  electromagnetic  spectrum.  MIL-HDBK  237D 
(Reference  (bk))  states  that  SS  must  be  given  appropriate  and  timely  consideration  in  acquisition 
planning,  development,  procurement,  and  deployment  of  spectrum-dependent  systems  or  equipment. 
Ensuring  compliance  with  national,  international,  and  DoD  policies  and  procedures  for  the  management 
and  use  of  the  electromagnetic  spectrum  is  critical  to  interoperability.  Both  E3  and  SS  must  be 
addressed  early  in  the  conceptual  phase  of  system  development  and  be  periodically  reviewed  and 
updated  throughout  the  system  design.  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  6212. OIF 
(Reference  (bl))  on  Net  Ready  Key  Performance  Parameter  (NR  KPP)  is  supportive  of  Reference  (bj ) . 
Reference  (bk)  provides  definitive  guidance  for  DoD  acquisitions.  Finally,  MIL-STD-461F  (Reference  (bm)) 
and  MIL-STD-464C  (Reference  (bn))  provide  details  on  technical  execution  of  these  requirements,  will 
prove  invaluable  as  references  for  the  E3  representative  on  design-related  IPTs  and  WIPTs. 

19.2  Key  Definitions  and  Concepts 

19.2.1  Responsibilities 

Responsibilities  of  the  DOT&E  and  DASD(DT&E),  OTAs,  PMs,  and  E3/SS  WIPT  are  delineated  in  Reference 
(bk)  and  are  outlined  below.  For  more  information,  see  Reference  (bk)  and  the  DOT&E  Guide  (Reference 
(bo)). 

19.2.1.1  DOT&E  AND  DASD  (DT&E) 

The  DOT&E  and  DASD  (DT&E)  shall: 

•  Approve  Service  TEMPs,  operational  test  plans,  early  assessments,  and  test  reports  on  T&E 
oversight  programs  (i.e.,  programs  on  the  OSD  T&E  oversight  list)  to  assess  the  adequacy  of 
E3/SS  testing. 
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•  Monitor  and  cite  E3/SS  issues  and  resolutions  during  participation  on  T&E  oversight  programs 
acquisition  IPTs. 

•  Review  Services  evaluation  approaches,  including  M&S,  small-scale  tests,  and  appropriate 
chamber,  field,  and  laboratory  tests. 

•  Review  spectral  characteristics  data  to  ensure  that  sufficient  information  is  available  for  test 
scenarios  and  to  support  the  resolution  of  E3/SS  issues. 

•  Report  the  status  of  E3/SS  issues  for  T&E  oversight  programs  in  the  DOT&E  Annual  Report,  and 
report  specific  program  findings  as  part  of  BLRIP  reports  to  the  SecDef  and  Congress. 

•  As  E3/SS  issues  related  to  fielded  systems  arise  during  OT&E  or  during  large-scale  training 
exercises  used  to  complement  operational  tests,  report  these  issues  to  the  appropriate  agencies 
for  resolution. 

19.2.1.2  OTAs  and  DT  Organizations 

The  OTAs  and  DT  Organizations  shall: 

•  Coordinate  E3/SS  assessments  during  field  tests  with  Services  E3/SS  points  of  contact  and  other 
DoD  agencies. 

•  Conduct  early  assessments  of  DoD  programs  to  identify  and  mitigate  potential  E3/SS  problems, 
including  hazards  of  electromagnetic  radiation  to  ordnance  (HERO). 

•  Include  E3/SS  and  spectrum  availability  assessment  issues  as  a  standard  presentation  at  OTRRs  . 

19.2.1.3  PM 

The  PM  shall: 

•  Ensure  that  E3/SS  T&E  efforts  receive  adequate  funding. 

•  Ensure  that  E3/SS  is  sufficiently  addressed  in  the  TEMP  because  it  will  receive  scrutiny  during 
the  TEMP  approval  process.  Ensure  that  E3/SS  requirements  are  addressed  from  the  outset  of 
the  program.  Requirements  should  be  addressed  in  Part  I  (Introduction),  and  known  or 
anticipated  test  issues  should  be  addressed  in  Part  III  (Test  and  Evaluation  Strategy).  If  a  need 
for  specialized  test  resources,  such  as  anechoic  chambers,  is  anticipated,  this  should  be  stated  in 
Part  IV  (Resource  Summary). 

•  Establish  the  E3/SSWIPT. 

19.2.1.4  E3/SS  WIPT 

An  E3/SS  WIPT  is  an  advisory  body  of  SMEs  that  may  be  established  by  the  PM  to  help  ensure  that  the 

system  under  development  has  spectrum  support  and  will  be  electromagnetically  compatible  with  itself 
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and  with  the  external  EME.  The  E3/SS  WIPT  should  be  organized  early  in  a  program  so  that  it  can 
contribute  to  the  trade-off  studies  of  alternate  concepts  and  to  assess  the  impact  of  design,  budgetary, 
and  scheduling  decisions  related  to  E3/SS  considerations.  The  E3/SS  WIPT  is  usually  composed  of 
Government,  Federally  Funded  Research  and  Development  Center,  and  development  contractor 
personnel  empowered  with  the  authority  to  make  most  decisions  within  their  discipline  while  being  held 
accountable  for  meeting  performance  and  cost  requirements.  The  team  is  expected  to  make  decisions  in 
a  cooperative  manner.  The  E3/SS  WIPT  should  be  co-chaired  by  a  Government  representative  and  a 
development  contractor  representative,  operating  under  the  authority  of  the  PM. 

Responsibilities  of  an  E3/SS  WIPT  should  be  defined  in  a  charter  and  may  include  the  following: 

•  Establishing  E3  performance  requirements  for  the  system  or  equipment,  by  drawing  from  and 
tailoring  existing  military  and  commercial  standards. 

•  Defining  the  flow  of  E3/SS  requirements  down  to  elements  of  the  system. 

•  Defining  and  updating  the  various  aspects  of  the  external  EME. 

•  Defining  E3/SS  requirements,  verification  methodology,  such  as  analysis,  M&S,  and  T&E. 

•  Participating  in  design  reviews  by  providing  technical  input  from  test  results  on 
system/subsystem  design  and  system  integration  relative  to  E3. 

•  Providing  inputs  to  system  specifications  on  E3. 

•  Providing  inputs  on  flight  clearance  requests. 

•  Preparing  and  updating  the  DD  Form  1494,  Application  for  Equipment  Frequency  Allocation,  for 
spectrum-dependent  systems  and  equipment. 

•  Defining  E3/SS  budget  requirements. 

•  Providing  E3/SS  inputs  to  acquisition  documents  and  reviewing  program  documentation  and 
contract  deliverables. 

•  Assessing  FIERO,  hazards  of  electromagnetic  radiation  to  personnel  (FIERP),  and  hazards  of 
electromagnetic  radiation  to  fuel  (FIERF)  safety  issues. 

•  Performing  E3  analyses  and  tests  to  identify  potential  E3/SS  problems  and  solutions. 

•  Identifying  operational  limitations  for  E3  problems  not  corrected. 

•  Evaluating  the  E3  impact  of  using  Commercial  Item/NDI  on  the  overall  performance  of  the  end 
item. 
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19.2.2  Role  of  Analyses,  Simulations,  and  Predictions 

Analyses  and  predictions  are  critical  in  identifying  and  resolving  potential  E3/SS  problems  during 
development;  therefore,  they  should  be  employed  as  early  in  a  program  as  possible,  before  there  are 
significant  expenditures  of  time,  effort,  and  money.  The  analyses  provide  essential  information  to  guide 
the  selection  of  appropriate  courses  of  action  to  correct  problems.  E3/SS  analyses  should  be  conducted 
and  continually  refined  throughout  the  items  life  cycle,  as  the  operational  EME  is  updated  and  as 
technical  characteristics  of  the  end-item  become  available.  These  analyses  are  typically  performed  at  an 
increasing  level  of  detail  during  each  stage  of  the  development  life  cycle.  As  measured  characteristics 
are  determined,  earlier  analyses  may  be  refined.  Available  test  data  for  interference  interactions  should 
also  be  fed  back  into  the  E3  analysis.  Careful  application  of  E3  analysis  and  prediction  techniques  at  the 
appropriate  phases  of  an  items  life  cycle  should  ensure  that  the  required  level  of  E3  control  is  defined 
without  having  the  wasteful  expense  of  over-engineering,  the  uncertainties  of  under-engineering,  or  the 
need  for  reengineering  at  an  ever  more  expensive  time  late  in  the  program. 

19.2.3  Key  Terms 

The  key  E3/SS  specific  terms  used  in  this  chapter  are  defined  in  Figure  19-1. 
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E3 

Electromagnetic 

Environment 

The  impact  of  the  EME  on  the  operational  capability  of  military  forces,  equipment,  systems,  and 
platforms.  It  encompasses  all  electromagnetic  disciplines,  discharge;  hazards  of  electromagnetic 
radiation  to  personnel,  ordnance,  and  volatile  including  EMC  and  electromagnetic  interference; 
electromagnetic  vulnerability;  electromagnetic  pulse;  electro-static  materials;  and  natural 
phenomena  effects  of  lightning  and  precipitation  static. 

EMC 

Electromagnetic 

Compatibility 

EMC  is  the  ability  of  a  system  to  operate  in  their  operational  EME  without  experiencing  or  causing 
unintentional  degradation  because  of  EM  radiation  or  response. 

EME 

Electromagnetic 

Environment 

The  EME  in  which  the  item  is  expected  to  operate  should  be  defined  early  in  the  acquisition 
process,  based  on  geographical  operating  area,  host  nation  constraints,  physical  operating 
environment  and  expected  interactions  with  friendly  and  hostile  systems. 

EMI 

Electromagnetic 

Interference 

EMI  is  any  EM  disturbance  -  intentional  (as  in  EW)  or  unintentional  -  degrades  or  limits  the  effective 
performance  of  electronic  equipment 

EMP 

Electromagnetic  Pulse 

EMP  is  the  non-kmizing  EM  radiation  (EMR)  from  a  nuclear  explosion. 

EMV 

Electromagnetic 

Vulnerability 

EMV  is  the  characteristic  of  an  item  that  causes  It  to  suffer  degraded  performance,  or  the  inability 
to  perform  its  specified  task,  as  a  result  of  the  operational  EME.  An  item  is  said  to  be  vulnerable  its 
performance  is  degraded  below  a  satisfactory. 

ESO 

Electrostatic 

Discharge 

ESO  can  cause  intermittent  or  upset  (transient)  failures  as  well  as  permanent  damage  to 
electronics  equipment.  ESO  can  also  cause  hazardous  conditions  in  fuels  and  ordnance,  as  well 
as  presenting  a  shock  hazard  to  personnel. 

HERE 

Hazards  of 
Electromagnetic 
Radiation  to  Fuels 

The  potential  for  electromagnetic  radiation  to  cause  ignition  or  detonation  of  volatile  combustibles 
is  referred  to  as  HERF  and/or  analyses. 

HERO 

Hazards  of 
Electromagnetic 
Radiation  to  Ordnance 

Hazards  that  result  from  adverse  interactions  among  radio  frequency  emitters  and  electrically 
initiated  devices  or  initiating  systems  contained  within  ordnance  systems  (e.g.,  fuses). 

HERP 

Hazards  of 
Electromagnetic 
Radiation  to  Personnel 

The  potential  for  electromagnetic  radiation  (EMR)  to  produce  harmful  biological  effects  in  humans 
is  referred  to  as  HERP. 

Lightning 

Lightning 

Lightning  effects  are  divided  into  direct  (physical  damage)  and  indirect  (EM-induced  electronics 
damage)  effects;  both  effects  may  occur  to  the  same  component. 

P-Static 

Precipitation  Static 

P-static  is  an  EM  disturbance  caused  by  a  random  electrostatic  charge  buildup  as  a  result  of  the 
flow  of  air,  moisture,  or  airborne  particles  over  the  structure  or  components  of  a  vehicle  moving  in 
the  atmosphere,  such  as  an  aircraft  or  spacecraft. 

SC 

Spectrum  Certification 

The  statement(s)  of  adequacy  received  from  authorities  of  sovereign  nations  after  their  review  of 
the  technical  characteristics  of  a  spectrum-dependent  equipment  or  system  regarding  compliance 
with  their  national  spectrum  management  policy,  allocations,  regulations,  and  technical  standards. 

SM 

Spectrum 

Management 

Planning,  coordinating,  and  managing  the  use  of  the  electromagnetic  spectrum  through 
operational,  engineering,  and  administrative  procedures  with  the  objective  of  enabling  electronic 
systems  to  perform  their  functions  in  the  intended  environment  without  causing  or  suffering 
unacceptable  interference. 

SS 

Spectrum 

Supportability 

The  assessment  as  to  whether  the  electromagnetic  spectrum  necessary  to  support  the  operation 
of  a  spectrum-dependent  equipment  or  system  during  its  expected  life  cycle  is,  or  wiH  be, 
available. 

Figure  19-1  Key  E3/SS  Terms  and  Definitions 
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19.3  E3/SS  Planning,  Analysis,  SE,  and  T&E  For  the  Systems  Acquisition  Process 


19.3.1  Planning  for  T&E  Considerations  throughout  the  Acquisition  Processes 


To  ensure  that  E3/SS  concerns  are  given  due  consideration,  it  is  essential  to  include  E3/SS  in  the 
acquisition  process  from  the  very  beginning.  The  following  sections  address  the  planning, 
documentation,  and  actions  in  which  E3/SS  must  be  taken  into  account  to  ensure  that  issues  have  been 
effectively  controlled,  and  that  limitations  and  vulnerabilities  have  been  identified  and  documented. 
These  evaluations  should  be  initiated  at  the  earliest  practical  point  (preferably  prior  to  MS  A)  in  the 
items  life  cycle  so  that  deficiencies  can  be  identified  early  and  corrected.  General  test  requirements  and 
guidelines  for  EMC  are  contained  in  Reference  (bm).  The  overarching  standard  is  Reference  (bn) .  The  S- 
Diagram  in  Figure  19-1  provides  gives  a  graphical  depiction  of  the  major  inputs  and  decision  points  in 
the  acquisition  process  from  an  E3/SS  perspective.  This  depiction  shows  decision  points  that  are  driven 
by  the  need  to  demonstrate  that  no  operational  E3  problems  will  exist  during  lOT&E.  Thus,  if  the 
consensus  of  experts  in  the  E3  field  is  that  there  is  no  possibility  of  E3  and  spectrum  issues  going 
forward,  then  from  that  point,  the  issue  is  no  longer  considered.  Detailed  guidance  is  available  in  section 
6  of  Reference  (bk).  The  following  sections  provide  a  brief  chronological  summary  of  the  documentation, 
technical  reviews,  and  test  activities  that  need  E3/SS  input.  The  checklists  for  each  milestone  are 
provided  at  the  end  of  this  chapter. 


Figure  19-2  S-Diagram 
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19.3.2  Requirements  Documentation 

Requirements  for  developmental  systems  are  often  determined  during  pre-acquisition  technology 
development,  such  as  the  warfighting  experiments  conducted  by  the  Military  Services  and  the  U.S.  Joint 
Forces  Command;  Advanced  Technology  Demonstrations  (ATDs);  and  ACTDs.  An  AoA  is  conducted  to 
demonstrate  preferred  concepts  that  will  go  forward  to  the  TD  phase  if  the  solution  requires  new 
development  rather  than  new  TTPs.  As  the  concept  for  the  technology  matures,  so  should  the 
understanding  of  the  EME  in  which  the  system  is  expected  to  operate.  Correctly  defining  the  EME  early 
in  the  program  will  have  a  significant  impact  on  success.  The  EME  includes  intra-platform  system 
interactions,  interaction  with  other  known  friendly  and  adversary  systems,  and  the  natural  and  man¬ 
made  environment  in  the  expected  area  of  operations. 

As  early  as  practical,  a  determination  should  be  made  as  to  whether  the  project  has  E3/SS  concerns  or 
safety  issues  regarding  hazards  of  electromagnetic  radiation  that  should  be  addressed.  The  E3/SS 
related  tasks  include  refining  cost  estimates  and  ensuring  that  all  of  the  needed  analysis  and  design 
work  is  planned.  Initial  analysis  needs  to  be  conducted  to  provide  assessments  sufficient  to  ensure  that 
E3/SS  requirements  are  adequately  addressed  in  the  appropriate  requirements  documents  in 
accordance  with  Reference  (ad) ,  ,  (bm)  and  (bn) . 

The  PM  must  ensure  that  E3/SS  requirements  are  addressed  from  the  outset  of  the  program.  This 
includes  both  performance  specifications  and  development  standards.  Detailed  guidance  is  provided  by 
MIL-HDBK  237  Series,  Reference  (bm)  and  (bn) .  The  ICD  should  define  authoritative,  measurable,  and 
testable  capabilities  and  define  KPPs.  The  information  support  plan  (ISP)  documents  the  programs 
interoperability  requirements  and  interface  requirements  and  must  include  consideration  of  E3/SS 
control.  The  ITR  should  support  the  development  and  clarification  of  these  requirements.  For  more 
details,  see  of  Reference  (bk) . 

It  is  highly  recommended  that  the  RFP  contain  language  that  clearly  makes  the  contractor  responsible 
for  ensuring  compliance  with  E3/SS  requirements.  It  is  important  to  realize  that  any  design  change  will 
require  reevaluation  of  the  E3  considerations.  Something  that  may  seem  trivial,  like  rerouting  a  cable  or 
moving  a  connector  6  inches,  can  cause  intra-system  electromagnetic  interference  (EMI)  problems.  EMI 
is  a  complex  problem  in  which  it  is  difficult  to  model  everything,  and  it  is  not  to  be  taken  lightly. 
Therefore,  the  contractor  is  best  able  to  monitor  this  level  of  detail  by  analyzing,  testing,  and  verifying 
components  and  systems  throughout  development.  For  example,  the  RFP  may  state  the  following: 

•  The  contractor  shall  design,  develop,  integrate,  and  qualify  the  system  such  that  it  meets  the 
E3/SS  performance  requirements  of  the  system  specification. 

•  The  contractor  shall  perform  analyses,  studies,  and  testing  to  establish  E3/SS  controls  and 
features  to  be  implemented  in  the  design  of  the  item. 

•  The  contractor  shall  perform  inspections,  analyses,  and  tests,  as  necessary,  to  verify  that  the 
system  meets  its  E3/SS  performance  requirements. 
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•  The  contractor  shall  prepare  and  update  the  DD  Form  1494  throughout  the  development  of  the 
system  for  spectrum-dependent  equipment  and  shall  perform  analysis  and  testing  to 
characterize  the  equipment,  where  necessary. 

•  The  contractor  shall  establish  and  support  an  E3/SS  WIPT  to  accomplish  these  tasks. 

•  IVIIL-HDBK-237D  may  be  used  for  guidance. 

The  SRR  is  a  system-level  review  conducted  to  ensure  that  the  EME,  E3/SS  performance  requirements, 
test  requirements,  and  plans  for  all  E3/SS  analyses  and  predictions  are  clearly  understood  by  both  the 
Government  and  the  contractor  and  are  spelled  out  in  the  SOW,  CDRLs,  the  specification,  and  the  TEMP 
as  needed. 

19.3.3  Develop  Systems  Specification 

19.3.3.1  Spectrum  Support 

0MB  Circular  No.  A-11  (Reference  (bp))  requires  that  spectrum  support  be  obtained  before  submitting 
funding  estimates  for  the  development  or  procurement  of  systems  or  equipment.  In  addition, 
certification  is  required  before  funds  are  obligated  for  spectrum-dependent  systems  or  equipment.  To 
accomplish  this  certification,  a  DD  Form  1494  must  be  submitted  to  the  appropriate  Service  frequency 
management  office  (FMO)  in  accordance  with  policies  and  procedures  of  DoDI  4650.01  (Reference  (bq)). 
Reference  (d) ,  and  the  form  itself.  The  data  required  and  provided  on  the  DD  Form  1494  are  maintained 
at  the  Joint  Spectrum  Center  and  benefit  that  portion  of  the  DoD  spectrum  management  (SM) 
community  involved  in  mission  planning  and  training  operations.  The  DD  Form  1494  must  be  updated  at 
each  design  review  to  ensure  that  the  data  are  current  and  accurate.  These  data  enable  the  following: 

•  Frequency  assignments  for  DoD  operations,  exercises,  and  training,  including  coordination  with 
foreign  (host)  nations  for  use  of  DoD  systems  overseas. 

•  Mitigation  or  resolution  of  EMI  problems. 

•  Siting  of  new  DoD  or  commercial  systems  on  ships,  on  aircraft,  in  space,  and  at  shore  sites. 

•  Integration  of  Cl  into  the  intense  EME  found  on  military  platforms  and  installations. 

•  Establishment  of  mutually  beneficial  parameters  for  spectrum  sharing  with  industry. 

The  PM  must  also  develop  a  justification  and  a  proposed  plan  to  obtain  SS  authorization  from  the 
National  Telecommunications  and  Information  Administration  or  the  Military  Communications- 
Electronics  Board  (MCEB)  to  proceed  into  the  EMD  phase. 

19.3.3.2  Establishing  IPTs 

Before  MS  B,  the  PM  must  establish  working  IPTs  as  needed  to  ensure  that  appropriate  SMEs  are 
involved  to  represent  the  relevant  technical  areas  well  enough  to  assess  program  risks  and  costs  related 
to  performance  trade-offs.  The  IPT  should  include  members  with  sufficient  expertise  in  electrical  and 
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electronics  engineering,  SM,  and  SE  to  address  E3/SS  issues  knowledgeably.  Later,  once  a  prime 
contractor  is  selected,  the  design  IPTs  (commonly  one  IPT  per  subsystem  and  one  overall  system  IPT) 
must  each  include  members  with  E3  expertise  to  guide  the  designers,  ensuring  compatibility  from  the 
earliest  design  stages  through  system  integration.  This  requirement  must  be  dictated  in  the  contract. 

19.3.3.3  TEMP 

The  TEMP  must  ensure  that  all  necessary  E3/SS  evaluations  are  conducted  during  DT&E  so  that  the 
assessments  performed  during  OT&E  will  not  find  new  unmitigated  E3  issues  such  as  performance  and 
operational  limitations  and  vulnerabilities.  Testing  is  a  shared  responsibility  of  the  contractor  and  the 
Government,  with  the  Government  bearing  the  responsibility  but  delegating  lower  level  testing  such  as 
unit  tests  and  most  integration  tests  to  the  contractor  and  holding  the  contractor  accountable  to 
document  the  test  methodology  and  results.  In  many  cases,  systems  will  include  both  DoD-developed 
and  COTS  electronic  equipment.  The  inclusion  of  CIs  and  NDIs  eases  the  design  work  but,  at  the  same 
time,  increases  the  importance  of  effectively  testing  for  E3  concerns  and  managing  spectrum  usage. 
Testing  validates  the  readiness  of  systems  to  be  fielded  into  the  environment. 

Content  requirements  for  the  TEMP  are  defined  in  Reference  (j) ,  and  details  are  found  in  References 
,  (bn)  and  (bo).  The  TEMP  is  organized  into  four  parts  that  should  address  the  following  issues: 

•  Introduction 

•  Are  MOEs  and  MOSs  established  for  E3/SS  requirements  that  are  addressed  in  the  CDD 
orCPD? 

•  Is  E3  identified  as  a  critical  operational  effectiveness  and  suitability  parameter? 

•  Are  MOEs  and  MOPs  stated  and  evaluation  criteria  and  data  requirements  defined  to 
include  E3/SS  considerations? 

•  Test  Program  Management  and  Schedule 

•  Is  the  schedule  for  E3  verification  events  identified? 

•  Is  T&E  responsibility  for  E3  verification  established  by  organization? 

•  T&E  Strategy 

•  Has  emission  and  susceptibility  testing  been  planned  for  the  subsystems  or  equipment 
in  accordance  with  Reference  (bm)  or  commercial  EMI  standards,  as  appropriate? 

•  Are  E3  tests  planned  for  Commercial  Item/NDI? 

•  Have  platform/system  E3  verifications  been  planned  in  accordance  with  Reference  (bn) 
?  (Note  that  EMI,  EMC,  and  electromagnetic  vulnerability  (EMV)  testing  should  be 
required  for  all  platforms  or  systems,  whereas  special  E3  T&E  efforts  such  as  HERO, 
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HERF,  HERP,  EMP,  lightning,  and  precipitation  static  (p-static)  may  be  required  on  a 
case-by-case  basis,  as  noted  in  the  CDD,  CPD,  TEMP,  or  contract  documents.) 

•  Are  E3/SS  issues  addressed  in  operational  test  plans? 

•  Have  intra-  and  inter-subsystem  and  equipment  E3  verifications  been  planned  for  an 
operational  environment? 

•  Have  intra-  and  inter-platform  and  system  E3  verifications  been  planned  for  an 
operational  environment? 

•  Are  special  E3  verifications  required,  depending  on  the  results  of  DT&E? 

•  Resource  Summary 

•  Have  adequate  resources,  including  M&S,  been  identified  for  the  following  efforts? 

•  Subsystem  and  equipment  emission  and  susceptibility  testing. 

•  Testing  of  CI/NDI. 

•  Reference  (bn)  verifications. 

•  Operational  intra-platform  and  system  EMI  evaluations. 

19.3.4  Design  Reviews 

19.3.4.1  SFR 

The  SFR  is  a  review  of  the  conceptual  design  and  technical  description  of  the  system  to  establish  its 
capability  to  satisfy  mission  requirements.  Both  developmental  (includes  contractor)  and  operational 
testers  should  be  reviewers  to  ensure  that  the  top-level  specifications  are  written  in  a  verifiable  manner 
and  that  the  test  program  is  sufficiently  robust  to  discover  and  mitigate  E3  issues. 

19.3.4.2  PDR 

The  PDR  confirms  that  the  preliminary  design  logically  follows  the  SFR  findings  and  meets  the 
requirements,  and  results  in  approval  to  begin  detailed  design.  E3/SS  documentation  and  test  plans 
must  be  updated  and  E3/SS  analysis  and  predictions  are  initiated.  Both  developmental  (includes 
contractor)  and  operational  testers  should  be  reviewers  to  ensure  that  the  mid-level  specifications  are 
written  in  a  verifiable  manner  and  that  the  test  program  is  sufficiently  robust  to  discover  and  mitigate  E3 
issues. 

19.3.4.3  CDR 

The  CDR  evaluates  the  completeness  of  the  design,  its  interfaces,  and  its  suitability  to  start  initial 
manufacturing.  E3/SS  inputs,  documentation,  and  test  plans  need  to  be  brought  up-to-date.  At  this 
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point,  it  is  necessary  to  ensure  that  the  design  has  taken  into  account  any  limitations  or  restrictions  on 
its  use  contained  in  the  approved  MCEB  DD  Form  1494  guidance  recommendations. 

19.3.4.4  TRR 

The  TRR  provides  the  assurance  that  the  system  has  undergone  a  thorough  test  process  and  is  ready  for 
turnover  to  the  next  test  phase.  Ensure  that  the  inputs  to  the  TRR,  the  E3/SS  analysis,  and  predictions 
have  been  completed  and  that  the  results  inform  the  criteria  for  the  TRR. 

19.3.4.5  E3/SS  Analyses  and  Predictions 

It  is  essential  that  E3/SS  analyses  and  predictions  be  employed  in  support  of  the  testing  and  evaluation 
of  electronic  platforms,  systems,  subsystems,  and  equipment.  E3/SS  analyses  are  critical  in  identifying 
and  resolving  potential  problems  during  development  and  ensuring  compatibility  in  the  developmental 
and  operational  test  phases  of  the  program.  The  analyses  provide  essential  information  to  guide  the 
selection  of  appropriate  courses  of  action  to  correct  problems  and  should  be  updated  and  presented  at 
each  design  review. 

19.3.5  Early  DT/OT  Assessments 

In  general,  much  of  the  testing  of  subsystems  and  initial  integration  testing  will  be  performed  by  the 
contractor's  system  developers  and  system  integrators  and  reported  to  and  reviewed  by  the  PM  and  the 
IPTs.  A  further  means  of  simplifying  the  DT&E  process  is  to  leverage  Commercial  Item/NDI,  often 
referred  to  as  COTS.  The  process  for  incorporating  these  items  is  discussed  in  section  6.7  of  Reference 
(bk).  However,  blindly  using  Commercial  Item  /NDI  carries  a  risk  of  E3/SS  problems  within  the  platform, 
system,  subsystem,  or  equipment.  Commercial  Item  /NDI  should  meet  the  operational  performance 
requirements  for  that  equipment  in  the  proposed  installation.  In  other  words,  a  Commercial  Item  /NDI 
may  indeed  be  certified  for  its  original  intended  use  but  when  integrated  into  a  new  system,  onto  a  new 
platform,  and/or  fielded  to  a  new  EME,  the  original  certification  assumptions  need  to  be  revisited  to 
determine  whether  a  new  evaluation  is  required. 

19.3.5.1  Subsystems  and  Equipment 

Reference  (bm)  and  section  6. 6. 3. 2. 2  of  Reference  (bj)  define  the  EMI  requirements  for  subsystems  and 
equipment  in  general;  that  is,  conducted  and  radiated  emission  and  susceptibility  (immunity).  Reference 
(bm)  also  describes  how  to  tailor  and  perform  tests  that  can  be  used  for  verification  of  these 
requirements. 

19.3.5.2  Platforms  and  Systems 

Reference  (bn)  and  section  6.6. 3. 2. 3  of  Reference  (bj)  define  the  developmental  E3/SS  requirements  for 
airborne,  sea,  space,  and  ground  platforms  and  systems  (both  new  and  modified),  including  associated 
ordnance.  Sectionl9.3.5.3  provides  an  overview  of  the  tests  needed  to  verify  these  requirements.  In 
general,  the  nature  of  these  tests  is  to  check  two  at  a  time  paired  sub-elements  to  see  whether  they 
interfere,  and  then  add  ever  increasing  elements  for  more  combinations.  This  testing  is  sometimes 
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referred  to  as  source/victim  testing.  The  limits  specified  in  Reference  (bm)  for  subsystems  and 
equipment  is  empirically  derived  to  cover  most  configurations  and  environments.  The  limits  have  a 
proven  record  of  success  demonstrated  by  the  relatively  low  incidence  of  problems  at  the 
platform/system  level. 

19.3.5.3  EMI  Testing 

19.3.5.3.1  Intra-platform  EMI 

EMI  requirements  are  a  risk-reduction  initiative,  and  adherence  to  them  will  afford  a  higher  degree  of 
confidence  that  the  platform  or  system  and  its  associated  subsystems  and  equipment  will  operate 
compatibly  upon  integration.  It  is  essential  within  a  platform  or  system  that  the  subsystems  and 
equipment  be  capable  of  providing  full  performance  along  with  other  subsystems  and  equipment  that 
are  operating  concurrently.  EMI  generated  by  a  subsystem  or  equipment  must  not  degrade  the  overall 
platform  or  system  effectiveness.  Intra-platform/system  EMI  is  one  of  the  basic  elements  of  concern  and 
is  addressed  in  detail  in  Reference  (o) . 

19.3.5.3.2  Inter-platform  EMI 

The  adverse  effects  of  electromagnetic  energy  from  one  system  on  another  or  from  radio  frequency 
emitters  in  the  operating  environment  are  well  documented.  Reference  (bn)  describes  airborne,  land- 
based,  ship-based,  air,  and  battlespace  EME  levels  and  addresses  the  requirement  for  inter¬ 
platform/system  EMI  in  detail.  In  addition,  MIL-HDBK-235-1C  (Reference  (br))  contains  friendly  and 
hostile  EME  levels,  as  well  as  emitter  characteristics. 

19.3.5.4  EMV 

Reference  (bq)  states  that  some  inter-platform/system  EMI  testing  may  be  performed  under  laboratory 
conditions,  in  which  the  item  under  test  and  the  simulated  EME  are  controlled.  Detection  of  undesired 
responses  during  routine  EMI  testing  might  necessitate  an  EMV  analysis  to  determine  the  impact  of  the 
laboratory-observed  susceptibility  on  operational  performance.  OT  in  the  actual  EME  rarely  is  effective 
in  the  investigation  or  verification  of  susceptibilities  because  there  is  much  less  control  on  variable 
conditions,  fewer  functions  can  generally  be  exercised,  and  expenses  can  be  high. 

19.3.5.5  Verification  of  Special  Requirements 

Systems  must  demonstrate  that  they  meet  required  protection  against  environmental  threats  to 
systems,  specifically,  p-static,  lightning,  EMP,  and  electromagnetic  radiation  hazards  (HERO,  HERP,  HERE) 
I  AW  References  ,  (bn)  and  (bo). 

19.3.6  Developmental  Performance  and  Verification  Testing 

DT&E  and  OT&E  are  to  be  conducted  at  each  successive  build  of  a  system:  the  EDM,  the  first  article 
acceptance  test,  the  LRIP  decision  point,  and  at  any  other  points  as  required. 
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To  understand  the  role  of  DT&E  and  OT&E  in  the  overall  acquisition  process,  it  is  worth  pointing  out  that 
a  properly  executed  DT&E  program  will  verify  that  system  design  and  integration  has  been  done 
according  to  design  and  will  prepare  the  system  for  operational  evaluation.  Per  Reference  (bk) ,  the 
OT&E  program  will  in  turn  do  the  following: 

•  Estimate  the  equipment  or  system  E3/SS  operational  effectiveness  and  operational  suitability, 
based  on  analyses  and  test  results  to  date. 

•  Test  in  the  operational  EME. 

•  Identify  needed  E3/SS  modifications. 

•  Provide  information  about  the  equipment  or  system  E3/SS  operational  performance,  including 
tactics,  doctrine,  organizational,  and  personnel  requirements. 

•  Verify  the  adequacy  of  supporting  E3/SS  documentation,  such  as  manuals,  handbooks,  support 
plans,  etc. 

•  Correct  and  retest  all  significant  E3/SS  deficiencies. 

After  MS  C,  system  changes  must  be  monitored  to  determine  their  impact  on  requirements  for  SS  and 
E3  control  and  to  update  appropriate  requirements  documents  as  necessary.  Changes  to  operational 
parameters  (e.g.,  tuning  range,  emission  characteristics,  antenna  gain  and  height,  bandwidth,  or  output 
power)  or  to  EME,  such  as  proposed  operational  locations,  may  require  additional  actions  through  an 
updated  DD  Form  1494  or  additional  E3  analyses  or  tests. 

19.3.7  Field  Assessments  (DT&E  in  the  field  in  preparation  for  lOT&E) 

19.3.7.1  Preparation  for  OT&E 

As  the  system  enters  development  and  operational  testing  cycles,  each  series  of  tests  must  be  preceded 
by  TRRs;  a  TRR  is  a  review  of  the  programs  readiness  to  begin  testing  at  any  level,  either  by  the 
contractor  or  by  the  Government.  See  Chapter  7  for  a  more  complete  description  of  the  TRR.  There  are 
several  types  of  readiness  reviews  that  may  apply.  Not  all  readiness  reviews  will  apply  to  each  system.  In 
general,  each  review  will  require  a  thorough  reevaluation  of  the  E3/SS  concerns,  with  updates  to 
requirements  documents  as  appropriate.  Also,  the  assignment  of  a  frequency  will  be  required  to 
conduct  DT  and/or  OT  in  open-air  ranges.  Testers  should  coordinate  with  range  managers  to  obtain 
frequency  assignment  well  in  advance  of  testing  . 

The  reviews  should  generate  specific  test  guidance  to  ensure  that  all  relevant  E3/SS  system  issues  are 
adequately  tested  and  reported.  Detailed  guidance  can  be  found  in  References  (bk) ,  (bm) ,  and  (bn). 

The  TRR  is  a  review  to  determine  readiness  for  system-level  testing,  both  DT  and  OT,  in  an  EME  with  the 
greater  realism  during  OT.  Recommended  E3/SS  actions  and  focus  areas  are  as  follows: 

•  Request  requisite  frequency  assignment(s). 


240 


525  &-yR9l3&lzuSya  l-yt-B^SyuDcms 


/KI-L0Jfi,o 


•  Verify  the  E3/SS  operational  test  plan  and  test  scenarios. 

•  Review  the  E3/SS  operational  effectiveness  and  suitability  thresholds  in  the  operational 
requirements  section  (Part  I)  of  the  TEMP. 

•  Validate  all  corrections  to  E3/SS  deficiencies  discovered  during  previous  testing. 

•  For  airborne  platforms  or  weapon  flight  tests  and  for  any  item  requiring  interaction  with  naval 
systems,  additional  flight  and  fleet  readiness  reviews  will  likely  be  required. 

As  further  preparation  for  testing,  ensure  that  adequate  organic  support  is  in  place  for  technical 
evaluation;  the  tests  should  be  planned  in  such  a  manner  that  they  can: 

•  Verify  attainment  of  E3/SS  performance  specifications  and  objectives. 

•  Demonstrate  that  E3/SS  risks  have  been  minimized. 

•  Evaluate  compatibility  and  interoperability  with  existing  or  planned  equipment  and  systems. 

•  Provide  assurance  that  the  equipment  or  system  is  ready  for  testing  in  the  operational  EME. 

It  is  possible  that  testing  at  this  stage  may  reveal  problems  that  require  operational  rather  than  materiel 
changes;  for  example,  prohibiting  use  of  certain  frequencies  or  modes  of  operation  in  some 
circumstances.  Any  such  work-arounds  or  changes  in  TTPs  must  be  part  of  the  ensuing  OT. 

19.3.8  Operational  Deployment 

19.3.8.1  Reviews  in  Support  of  Production  Decision 

•  SVR  and  PRR.  Once  OT&E  is  complete,  the  SVR  is  conducted  to  verify  that  the  production 
configuration  complies  with  the  performance  specification.  The  PRR  is  conducted  prior  to  any 
production  decision  to  validate  design  readiness,  resolution  of  production  engineering 
problems,  and  accomplishment  of  production  phase  planning.  Recommended  E3/SS  actions  are 
as  follows: 

o  Validate  all  corrections  to  E3/SS  deficiencies  discovered  during  previous  testing, 
o  Review  and  approve  E3/SS  test  reports. 

o  Ensure  that  subsequent  procurements  and  replacement  parts  meet  original  E3/SS 
program  requirements. 

o  Physical  Configuration  Review  (PCR).  The  PCR  formalizes  the  product  baseline,  including 
specifications  and  the  technical  data  package,  so  that  future  changes  can  be  made  only 
through  full  configuration  management  procedures. 
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19.3.8.2  OT&E 


The  Services  OTAs  perform  E3/SS  OT&E  to  evaluate  operational  performance  in  the  presence  of  other 
operating  items  and  compliance  with  KPPs  described  in  Part  I  of  the  TEMP  and  to  identify  any  resulting 
limitations  and  vulnerabilities.  OT&E  should  be  accomplished  in  as  realistic  an  operational  EME  as 
possible.  It  is  important  that  resources  and  assets  required  for  verification  of  E3  requirements  be 
identified  early  in  the  program  to  ensure  their  availability  when  needed.  The  EME  and  conduct  of  tests 
should  be  addressed  at  a  high  level  in  Part  III  of  the  TEMP.  The  following  guidance  applies  to  operational 
E3  testing: 

•  Items  used  for  verification  should  be  in  the  up-to-date  production  configuration,  preferably  the 
first  article. 

•  EMI  qualification  testing  to  either  References  (bm)  or  (bn) ,  as  applicable,  should  be  performed 
before  OT. 

•  All  items  should  be  placed  in  modes  of  operation  that  will  maximize  potential  indications  of 
interference  or  susceptibility,  consistent  with  overall  operational  performance  requirements. 

•  If  it  has  been  determined  during  DT  or  field  tests  that  work-arounds  or  changes  in  TTPs  are 
needed,  the  OT  personnel  need  to  demonstrate  that  they  can  adapt  to  the  changes  and  the 
changes  must  be  deemed  to  be  operationally  satisfactory. 

19.3.8.2.1  Intra-Platform/System  EMI  Testing 

Subsystems  and  equipment  should  be  designed  and  integrated  to  coexist  and  to  provide  the  operational 
performance  required  by  the  user.  The  following  issues  should  be  addressed  during  operational  intra¬ 
platform/system  EMI  testing: 


•  Potential  EMI  source  vs.  victim  pairs  should  be  identified  and  systematically  evaluated  by 
exercising  the  subsystem  and  equipment  onboard  the  platform  or  system  through  the  various 
modes  and  functions  while  monitoring  the  remaining  items  for  degradation.  Both  one  source  vs. 
one  victim  and  multiple  sources  vs.  one  victim  conditions  should  be  evaluated. 

•  Evaluation  of  transmitters  and  receivers  should  be  conducted  across  their  entire  operating 
frequency  ranges. 

•  Testing  should  be  conducted  in  an  area  in  which  the  ambient,  or  background,  EME  does  not 
affect  the  validity  of  the  test  results. 

•  Testing  should  include  all  relevant  external  hardware  such  as  weapons,  stores,  provisioned 
items  (i.e.,  those  installed  in  the  platform/system  by  the  user)  and  support  equipment. 
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19.3.8.2.2  Additional  Intra-Ship  Concerns 

19.3.8.2.2.1  Intermodulation  Interference 

The  large  number  of  high-frequency  transmitters,  their  high  output  power,  and  the  construction 
techniques  and  materials  used  on  modern  ships  make  the  presence  of  intermodulation  interference  a 
reality.  On  surface  ships,  various  currents  from  the  different  transmitters  mix  in  non-linearities  within 
the  hull  to  produce  signals  at  sums  and  differences  of  the  fundamental  and  harmonic  frequencies  of  the 
incident  signals.  Reference  (bn)  specifies  the  tests  and  analyses  necessary  to  verify  compliance  with  the 
applicable  internal  EME  requirements. 

19.3.8.2.2.2  Shipboard  HERO  and  EME  Surveys 

Procedures  are  implemented  onboard  Navy  ships  to  protect  ordnance  from  the  effects  of  the  EME 
generated  by  high-power  shipboard  transmitters.  These  procedures  include  creating  a  ship-specific 
HERO  emission  control  instruction.  A  survey  should  be  performed  to  identify  transmitters,  antennas,  and 
ordnance  handling  and  loading  areas  throughout  the  ship.  After  the  susceptibility  characteristics  of  the 
ordnance  are  ascertained,  the  ships  operational  worst-case  EME  should  be  determined  to  ensure  that 
potentially  hazardous  EME  levels  are  not  present  in  areas  in  which  ordnance  may  be  stored,  handled,  or 
used.  Emissions  from  transmitters  capable  of  producing  potentially  hazardous  EMEs  should  then  be 
measured  in  worst-case  conditions  at  all  ordnance  locations. 

19.3.8.2.3  Inter-Platform/System  E3  Evaluations 

As  with  intra-system  testing,  interactions  between  known  systems  raise  issues  that  should  be  addressed 
during  operational  inter-platform/system  E3  evaluations,  both  testing  and  analyses.  Consider  the 
following: 

•  Potential  EMI  source  vs.  victim  pairs  from  friendly,  hostile  (if  known),  joint,  and  combined  forces 
should  be  identified  and  systematically  evaluated  by  exercising  the  subsystems  and  equipment 
on  each  platform  and  system  through  their  various  modes  and  functions  while  monitoring  the 
remaining  items  for  degradation.  Both  one  source  vs.  one  victim  and  multiple  sources  vs.  one 
victim  conditions  should  be  evaluated. 

•  A  frequency  selection  or  emission  control  plan  should  be  developed  for  evaluation  of 
transmitters  and  receivers  across  their  entire  operating  frequency  range. 

•  Operational  evaluation  of  responses  identified  by  M&S  should  be  performed. 

•  Testing  should  be  conducted  in  an  area  and  at  a  time  in  which  the  ambient,  or  background,  EME 
does  not  affect  the  validity  of  the  test  results. 

•  Testing  should  include  all  relevant  external  hardware  such  as  weapons,  stores,  provisioned 
equipment  (i.e.,  items  installed  in  the  platform  or  system  by  the  user),  and  support  items. 
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19.3.8.2.4  Additional  Ordnance  Concerns 

Inter-platform/system  E3  testing  involving  ordnance  should  include  preflight,  captive-carry,  and  free- 
flight  configurations  of  the  ordnance.  Free-flight  testing  of  ordnance  may  be  simulated  utilizing  an  inert, 
instrumented,  ordnance  device  suspended  in  a  quiet,  EM-free  environment,  such  as  an  anechoic 
chamber.  Use  of  the  anechoic  chamber  is  recommended  to  determine  the  radio  frequency  (RF)  points 
and  aspect  angles  associated  with  specific  susceptibilities.  The  free-flight  test  program  consists  of 
evaluating  weapon  performance  during  the  launch,  cruise,  and  terminal  phases  of  flight,  while  exposed 
to  friendly  and  hostile  EME. 

19.3.8.2.5  Additional  Aircraft  Concerns 

The  EME  that  may  be  encountered  by  an  aircraft  must  be  reviewed  and  the  status  of  the  aircraft  with 
regard  to  the  environment  must  be  evaluated  prior  to  flight.  EMI  testing  of  the  subsystems/equipment 
can  be  used  as  a  baseline  of  hardness.  Flowever,  limited,  inter-platform/system  testing  involving  specific 
emitters  may  be  necessary.  If  such  tests  are  not  performed,  operational  restrictions  on  flight  paths  may 
need  to  be  imposed. 

19.4  Summary 

E3  and  spectrum  concerns  are  challenges  of  the  modern  system.  Proper  SE,  informed  by  thorough  M&S 
and  even  more  thorough  T&E,  is  required  for  successful  system  deployment  and  utilization.  The  key 
points  in  the  system  development  life  cycle  that  specifically  require  E3/SS  input  are  summarized  in  the 
checklists  below. 

CHECKLIST  FOR  T&E  ASSESSMENTS  FOR  E3  and  SS 

Objective:  To  identify  to  the  best  extent  possible  the  E3  and  SS  limitations  and  vulnerabilities  of  the 
subject  system. 

PM  Responsibilities: 

1.  DD  Form  1494  submitted  to  the  Service  FMO. 

2.  Status  of  host-nation  frequency  supportability  description  of  operational  EME  (e.g.,  operational 
environment,  theater,  mission  in  the  operations  plan). 

3.  Latest  program  documentation  (e.g.,  ICD,  CDD,  CPD,  APB,  C4  ISP,  specification). 

4.  TEMP  that  contains: 

a.  E3  within  the  scope  of  a  COI  (TEMP,  Part  I). 

b.  List  of  tests  and  analyses  used  to  determine  the  equipment  effectiveness/  suitability/survivability 
performance  in  the  operational  EME  (TEMP,  Part  III). 

5.  Copy  of  the  following  analyses  and/or  T&E  data: 
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a.  Intra-platform/system  analyses: 

(1)  Antenna  coupling  and  blockage  analyses  and/or  test  data. 

(2)  Subsystem/equipment  EMC  analyses  and/or  test  data. 

(3)  CI/NDI/GFE  EMC  analyses  and/or  test  data. 

b.  Inter-platform/systems  EMC  analyses  and/or  test  data  for  spectrum-dependent  (Joint  E3  Evaluation 
Tool  (JEET)  model)  and  non-spectrum-dependent  equipment. 

6.  Special  E3  analyses  and/or  test  data  (i.e.,  HERO,  HERP,  HERE,  EMP,  lightning,  and  p-static),  if 
required  by  the  CDD,  CPD,  or  TEMP. 

7.  E3  and  SM  impact  assessments  that  identify  and  define  operational  limitations  and 
vulnerabilities  (i.e.,  lessons  learned). 

8.  DT&E  test  plans  and  results/reports. 

Lead  DT  and  OTA  Responsibilities: 

1.  Develop  and  execute  DT&E  test  plans  and  evaluate  results  in  advance  of  OT&E. 

2.  Develop  and  execute  OT&E  test  plan  and  evaluate  results. 

3.  Evaluate  spectrum  certification  documents  and  comply  with  requirements  as  appropriate. 

ACQUISITION  DATA  REQUIREMENTS  CHECKLISTS  FOR  E3  and  SS 
Milestone  A  Checklist 

Objective:  To  ensure  that  E3  and  SS  are  addressed  in  requirements  documents,  ACTDs/ATDs,  joint 
warfighting  experiments,  concept  refinements,  and  technology  developments. 

Required  Information: 

1.  DD  Form  1494  submitted  to  Service  FMO  indicating  intent  to  comply  with  applicable  DoD,  national, 
and  international  SM  policies  and  regulations. 

2.  Requirements  documents,  demonstrations,  and  experiments  that  address  the  following: 

a.  Does  the  project  address  E3? 

b.  Does  the  project  address  the  requirement  for  SS? 

c.  Does  the  project  address  the  safety  issues  regarding  HERO,  if  applicable? 

Milestone  B  Checklist 
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Objective:  To  ensure  that  E3  and  SS  issues  are  identified  and  adequately  addressed. 

Required  Information: 

1.  DD  Form  1494  submitted  to  Service  FMO. 

2.  Status  of  host-nation  approval  (FINA)  efforts. 

3.  CDD/CPD  that  addresses  the  following: 

a.  Description  of  operational  EME  (the  operational  environment,  theater,  mission  in  the  operations  plan, 
etc. 

b.  Compliance  with  applicable  DoD,  national,  and  international  SM  policies  and  regulations. 

c.  Intra-  and  inter-platform/system  EMC. 

d.  E3  specialty  issues  (FIERO,  FIERP,  FIERF,  EMP,  lightning,  electrostatic  discharge  (ESD),  and  p-static,  as 
appropriate). 

4.  TEMP,  with  a  set  of  KPPs  (TEMP,  Part  I),  a  list  of  verification  efforts  that  addresses 
effectiveness/suitability/  survivability  of  the  platform,  system,  subsystem,  or  equipment  in  the  intended 
operational  EME  and  provisions  for  testing  CI/NDI  (TEMP,  Part  III). 

5.  E3  and  SS  potential  risks  identified  and  tests  and  analyses  performed  to  date  that  identify  and  define 
operational  limitations  and  vulnerabilities. 

NOTE:  All  acquisition  documents  should  contain  requirements  for  E3  and  SS  tests  and  analyses. 

Milestone  C  Checklist 

Objective:  To  identify  to  the  best  extent  possible  E3/SS  limitations  and  vulnerabilities  of  the  subject 
system,  subsystem,  or  equipment. 

Required  Information,  as  appropriate  : 

1.  DD  Form  1494  submitted  to  Service  FMO. 

2.  Status  of  FINA  effort. 

3.  Description  of  operational  EME  (i.e.,  the  operational  environment,  theater,  mission  in  the  operations 
plan,  etc). 

4.  Latest  program  documentation  (mission  area  ICD,  CDD,  CPD,  ISP,  performance  specification,  and 
SOW). 

5.  TEMP  that  contains: 
a.  KPPs  (TEMP,  Part  I). 
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b.  A  list  of  tests  and  analyses  used  to  determine  the  items  effectiveness/  suitability/survivability  in  the 
operational  EME  (TEMP,  Part  III). 

6.  Copies  of  the  following  verification  results,  including  T&E  data: 

a.  Intra-platform/system  data,  including: 

(1)  Antenna  coupling  and  blockage  analyses  and  test  data. 

(2)  Subsystem/equipment  EMC  analyses  and  test  data. 

(3)  CI/NDI  EMC  analyses  and  test  data. 

b.  Inter-platform/system  EMC  verification  results  and  test  data  for  spectrum-dependent  (JEET  model) 
and  non-spectrum-dependent  equipment). 

c.  Special  E3  analyses  and  test  data  (HERO,  HERP,  HERE,  EMP,  lightning,  ESD,  and  p-static)  if  required  by 
the  CDD,  CPD,  TEMP,  or  contractual  documents. 

7.  E3  and  SM  impact  assessments  that  identify  and  define  operational  limitations  and  vulnerabilities, 
including  lessons  learned. 

8.  DT&E  test  plans  and  results/reports. 

9.  Initial  OT&E  test  plans  and  results/reports. 

10.  User-initiated  test  results. 
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Test  &  Evaluation  Management  Guide 

Chapter  20  -  Special  Topics:  Interoperability  and  Cyber  Security 

20.1  Policy 

The  interoperability  and  cyber  security  requirements  for  information  systems  and  national  security 
systems  are  described  in  several  DoD  policy  documents  including  the  following: 

•  DoDI  4630.8  (Reference  (bs)). 

•  Reference  (bq). 

•  DoDD  8500.01E  (Reference  (bt)). 

•  DoDI  8500.2  (Reference  (bu)). 

•  DOT&E  Memorandums  (Reference  (bv)  and  (bw) ). 

The  Joint  Staff  also  addresses  interoperability  and  cyber  security  in  Reference  (ad) ,  (bl) ,  CJCSI  6212 
Series  Interoperability  and  Supportability  of  Information  Technology  and  National  Security  Systems,  and 
CJCSI  6510.01F  (Reference  (bx)).  DoD  policy  documents  pertaining  to  cyber  security  and  the  DoD  Risk 
Management  Framework  are  being  drafted.  These  new  documents  as  well  as  updates  to  existing 
documents  will  impact  T&E  support..  The  latest  version  of  the  referenced  document  should  be 
referenced  when  considering  interoperability  and  cyber  security.  The  above  references  are  mutually 
supportive  and  collectively  can  be  used  to  understand  and  shape  T&E  to  support  interoperability  and 
cyber  security.  Additional  T&E  guidance,  provided  by  DOT&E,  related  to  cyber  security  and 
interoperability  is  provided  in  References  (bw)  and  (bx). 

Early  involvement  by  the  T&E  community  with  the  PM,  security  certification  and  accreditation  agents, 
and  the  JITC  is  essential.  This  involvement  may  include  collaborative  test  planning  and  execution 
conducted  to  integrate  T&E  activities  such  that  test  data  can  be  shared  and  reused  to  support  multiple 
stakeholders. 

20.2  Interoperability  Testing 
20.2.1  NR  KPP 

Interoperability  is  measured  by  the  NR  KPP,  which  is  used  to  assess  information  needs,  information 
timeliness,  cyber  security,  joint  interoperability  and  supportability,  and  attributes  required  for  technical 
exchange  of  information.  The  NR-KPP  is  described  in  both  DoD  issuances  and  Chairman  of  the  Joint 
Chiefs  of  Staff  directives.  Reference  (bl)  is  used  to  assess  information  needs,  information  timeliness, 
cyber  security,  and  net-ready  attributes  required  for  both  the  technical  exchange  of  information  and  the 
end-to-end  operational  effectiveness  of  that  exchange.  The  NR  KPP  consists  of  verifiable  performance 
measures  and  metrics  required  to  evaluate  the  timely  and  accurate  exchange  and  utility  of  information 
to  satisfy  information  needs  for  a  given  capability. 
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The  underlying  concept  driving  the  inception  of  the  NR-KPP  is  the  DoDs  move  to  a  net-centric 
environment  (NCE).  The  NCE  is  a  framework  for  full  human  and  technical  connectivity  and 
interoperability  that  allows  DoD  users  and  mission  partners  to  share  the  information  they  need,  when 
they  need  it,  in  a  form  they  can  understand  and  act  on  with  confidence,  and  protects  information  from 
those  who  should  not  have  it.  The  NCE  supports  user  requirements  for  seamless  information  sharing, 
ubiquitous  application  reach,  and  a  fully  collaborative  environment  across  organizations  and  domains. 
The  NR-KPP  focuses  on  the  information  sharing  and  security  capability  of  the  NCE.  The  strategy  for 
achieving  net  readiness  should  be  documented  in  the  TEMP. 

The  inclusion  of  the  NR-KPP  in  CDDs,  CPDs,  and  ISPs  is  mandatory  for  all  acquisition  programs  and  post¬ 
acquisition  programs  for  systems  used  to  enter,  process,  store,  display,  or  transmit  DoD  information, 
regardless  of  classification  or  sensitivity. 

Throughout  the  program  life  cycle,  the  NR  KPP  is  used  for  analyzing,  identifying,  and  describing  system 
interoperability  requirements  and  developing  test  strategies  to  assess  them.  The  test  strategy  should  be 
specified  to  a  level  of  detail  that  allows  verification  of  interoperability. 

Final  NR-KPP  certification  is  routinely  assessed  in  conjunction  with  lOT&E  but  is  documented  separately 
through  the  JITC. 

20.2.2  NR-KPP  Compliance 

Reference  (bl)  calls  for  the  identification  of  the  NR  KPP  via  a  three-step  process  of  mission  analysis, 
information  analysis,  and  SE.  The  specific  elements  of  NR-KPP  compliance  are  no  longer  specified  in 
Reference  (bl) ;  however,  common  elements  are  defined  in  Reference  (j) . 

NR  KPP  compliance  encompasses  five  elements  that  should  be  included  for  NR-KPP  certification. 
Simplified  summaries  of  these  elements  are  provided  below: 

Element  1  Required  Documentation  .  Having  the  required  documentation  means  that  means  that 
selected  aspects  of  the  systems  architectural  products  are  compliant  as  documented  via  the  DoD 
Architecture  Framework  (DoDAF)  (Reference  (cb))  with  mandated  technical  interface  standards,  that 
required  interface  and  data-sharing  standards  are  being  implemented,  that  cyber  security  requirements 
to  include  system  accreditation  are  being  considered,  and  that  applicable  SS  documentation 
requirements  are  being  met.  From  a  T&E  perspective,  interoperability  requirements  are  confirmed  in 
the  approved  capabilities  document,  and  ISP/tailored  ISP.  The  cyber  security  requirements  are  generally 
described  in  the  capabilities  document,  but  definitive  cyber  security  requirements  can  be  confirmed  in 
the  cyber  security  acquisition  strategy  and/or  RMF  package  and  provide  the  basis  for  cyber  security  RMF 
certification  and  accreditation.  Spectrum  certification  is  documented  in  the  DD  Form  1494/JF-12  and/or 
Note  to  Holder.  Notes  to  Holders  of  DD  Form  1494s  provide  a  process  to  update  existing  DD  Form  1494 
forms  as  operational  systems  undergo  modifications  which  change  the  operating  parameters  of  a 
spectrum-dependent  device. 
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Element  2  Supporting  Integrated  Architecture  Products  .  This  element  ensures  that  the  products  exist 
and  are  developed  in  accordance  with  DoDAF  standards  and  that  they  sufficiently  describe  a  net-centric 
environment  consistent  with  the  capability  needs  of  the  program.  From  a  T&E  perspective,  the  key 
integrated  architecture  viewpoints  include  the  Technical  Standards,  Systems  Viewpoint  (SV)-l  Systems 
Interface  Description,  SV-6  Systems  Resource  Flow  Matrix,  and  Operational  Viewpoint  (OV)-6c  Event- 
Trace  Description. 

Element  3  Global  Information  Grid  (GIG)  Technical  Guidance  Compliance  .  This  element  describes  the 
activities  required  to  establish,  use,  operate,  and  manage  the  DoD  net-centric  enterprise  information 
environment  to  include  the  generic  user  interface,  intelligent-assistant  capabilities,  net-centric  service 
capabilities  (core  services,  community-of-interest  services,  and  environment  control  services),  and 
enterprise  management  components.  It  also  describes  a  selected  set  of  key  standards. 

The  GIG  is  the  globally  interconnected,  end-to-end  set  of  information  capabilities,  associated  processes, 
and  personnel  for  collecting,  processing,  storing,  disseminating,  and  managing  information  on  demand 
to  Warfighters,  policy  makers,  and  support  personnel.  The  GIG  includes  all  of  the  DoDs  owned  and 
leased  communications  and  computing  systems  and  services,  software  (including  applications),  data, 
security  services,  and  other  associated  services  necessary  to  achieve  information  superiority. 

The  GIG  Technical  Guidance  defines  the  objective  end-state  for  the  GIG,  which  provides  a  wide  variety  of 
on-demand  services  to  users  and  is  the  centerpiece  of  the  DoDs  move  to  an  NCE.  From  a  T&E 
perspective,  compliance  with  this  requirement  is  based  upon  the  approved  capabilities  document  and 
ISP.  Additional  T&E  is  not  required. 

Element  4  Compliance  with  DoD  Cyber  Security  Requirements  .  Cyber  security,  formerly  identified  as 
lA,  includes  measures  that  protect  and  defend  information  and  information  systems  by  ensuring  their 
availability,  integrity,  authentication,  confidentiality,  and  non-repudiation.  Key  components  of  the  DoD 
Cyber  Security  Program  include  the  following: 

•  Risk  Management .  Management  of  IT  risks  to  protect  U.S.  interests;  the  Departments  ability  to 
conduct  operations;  and  DoD  individuals,  organizations,  and  assets  in  accordance  with  the 
National  Institute  of  Standards  and  Technology  (NIST)  Special  Publication  (SP)  (Reference  (by)). 

•  Operational  Resilience  .  Developing  and  fielding  systems  that  are  flexible,  adaptable,  and 
successful  in  the  face  of  cyber  degradation,  loss,  or  attack. 

•  Integration  and  Interoperability 

o  Semantic  Interoperability.  The  ability  of  each  sending  party  to  communicate  data  and 
have  receiving  parties  understand  the  message  in  the  sense  intended  by  the  sending 
party. 

o  Technical  Interoperability.  The  ability  for  different  technologies  to  communicate  and 
exchange  data  based  upon  well-defined  and  widely  adopted  interface  standards. 
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o  Policy  Interoperability.  The  common  business  processes  related  to  the  transmission, 
receipt,  and  acceptance  of  data  among  participants. 

•  Cyber  Defense  .  Ensuring  that  systems  have  the  capability  to  take  actions  to  protect,  monitor, 
analyze,  detect,  and  respond  to  unauthorized  activity  within  DoD  information  networks. 

From  a  T&E  perspective,  minimum  cyber  security  RMF  requirements  are  dependent  upon  the  mission 
assurance  category  (MAC)  and  confidentiality  level  (CL)  as  defined  in  the  References  (bt)  and  (bu)  and  as 
documented  in  the  lA  acquisition  strategy  and  RMF  package.  Some  lA  controls  will  be  required  by  the 
system  under  test  and  some  will  be  inherited  from  the  hosting  enclave  and  the  computer  network 
defense  service  provider.  References  (bv)  and  (bw)  describe  a  six-step  process  that  is  heavily  dependent 
upon  lA  controls  verification  activities  executed  prior  to  OT&E.  Early  T&E  stakeholder  involvement 
including  the  DT  agent,  security  certification  and  authorization  agent,  and  OT  agent  to  execute 
integrated  test  planning  and  execution  will  facilitate  test  data  sharing  and  streamline  certification,  DT, 
and  promote  successful  lA  OT.  lA  controls  implemented  by  the  system  and  supporting  enclave  must  be 
in  place  to  enable  valid  interoperability  testing.  lA  and  cyber  security  requirements  are  described  in 
section  20.6  of  this  guide. 

Element  5  Compliance  with  SS  .  This  element  involves  compliance  with  national,  international,  and  DoD 
policies  and  procedures  for  the  management  and  use  of  the  electromagnetic  spectrum.  From  a  T&E 
perspective,  the  spectrum  certification  is  confirmed  by  inspecting  the  DD  Form  1494/JF-12  and/or  Note 
to  Flolder.  As  described  in  Reference  (bq)  the  spectrum  certification  process  begins  prior  to  MS  A.  A 
Stage  3DD  Form  1494  is  typically  required  in  advance  of  MS  B.  A  Stage  4  DD  Form  1494  is  typically 
required  in  advance  of  MS  C  . 

20.3  "Operationalizing"  the  NR  KPP 

The  goal  for  the  OT&E  community  is  to  design  a  test  plan  for  the  NR  KPP  that  is  operationally 
meaningful;  that  is,  a  test  plan  that  evaluates  whether  the  system/service/capability  enables  a  typical 
operator  to  do  the  following,  in  support  of  the  mission  without  significantly  increasing  the  burden  on 
the  operator,  system  administrator,  or  maintainer: 

•  Enter  and  manage  data  in  the  network  (can  be  configured  and  managed  as  a  network  entity). 

•  Exchange  data  (preserves  the  meaning  and  relationships  of  the  information  exchanged,  given 
the  network  constraints,  within  performance  thresholds,  in  a  secure  manner  (meets  lA 
attributes))  with  appropriate  spectrum  considerations. 

•  Continue  to  have  access  to  critical  mission  functions  when  under  attack  (exhibiting  operational 
resilience  for  critical  mission  operations). 

A  test  design  must  focus  on  the  mission  contribution  of  the  system  or  service  to  mission 
accomplishment.  Various  methodologies  can  be  used  to  assist  the  test  design  including  mission  threads, 
functional  decomposition,  and  mission  execution. 
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A  mission  thread  focuses  the  exchange  of  information  for  sensor  to  shooter  (i.e.,  from  source  through 
fusion  to  end  user).  Functional  decomposition  and  data  collection  directly  assess  the  contribution  of  the 
new  capability  in  terms  of  task  performance,  collaboration,  and  overall  mission  performance. 

Data  collection  occurs  at  key  interfaces  (i.e.,  collaboration,  work-arounds,  and/or  bottleneck),  using 
instrumentation  where  available,  in  conjunction  with  SME  observations.  Use  of  mission  threads  to  trace 
information  exchange  and  functional  decomposition  to  identify  the  tasks  performed  along  the  thread 
assesses  the  contribution  of  the  system  in  terms  of  task  performance,  collaboration,  and  overall  mission 
performance. 

In  designing  a  test  plan,  consider  the  following: 

•  Look  for  a  high  degree  of  organization-to-organization  interaction  and  information 
exchange/sharing  (collaboration). 

•  Look  for  broad  coverage  spanning  multiple  echelons  (depth)  and  multiple  mission/functional 
areas  (breadth). 

•  Look  for  high  concentration  of  information  systems  (integration). 

•  Look  for  dependency  on  information  systems. 


20.4  Role  of  JITC 

For  all  systems,  JITC  is  responsible  to  the  Joint  Staff  Directorate  for  Force  Structure,  Resource,  and 
Assessment  (J-8)  to  provide  interoperability  test  certification.  JITC  interoperability  evaluations  will  assess 
the  degree  to  which  the  NR-KPP  requirements  are  met  and  the  expected  operational  impact  of  any 
observed  discrepancies.  JITC  may  function  as  the  lead  testing  agency  or  use  test  data  from  other  sources 
to  evaluate  interoperability.  Validation  of  interoperability  cannot  be  accomplished  in  conjunction  with 
the  systems  operational  assessment  alone;  as  with  other  testing  and  certification  agencies,  JITC  must  be 
engaged  early  and  requires  adequate  funding  from  the  PM. 

The  JITC  NR-KPP  evaluation  will  leverage  all  available  test  activities  to  achieve  an  interoperability 
certification  in  an  efficient  and  effective  manner.  JITC  may  use  test  data  provided  by  the  contractor  to 
evaluate  standards  conformance,  data  from  DT  agents  to  verify  Systems/Service  Data  Exchanges 
demonstrated  in  DT,  and/or  data  from  the  OT  agent  to  validate  Operational  Information  Exchanges 
demonstrated  during  OT.  Generally,  a  special  test  will  be  conducted  by  the  JTIC  only  if  other  test  data 
are  not  available.  Therefore,  it  is  important  to  involve  the  JITC  as  a  key  participant  in  test  planning 
efforts.  It  is  critical  that  test  organizations  establish  an  early  framework  and  test  methodology  as  a 
foundation  for  coordinating  their  activities  with  the  JITC  to  ensure  that  each  of  the  elements  described 
in  section  20.2.2  is  properly  assessed. 
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20.5  Supporting  Integrated  Architecture  Products 

The  supporting  integrated  architecture  products  (Element  2  cited  in  section  20.2.2)  are  a  documented 
collection  of  different  architectural  perspectives  or  viewpoints.  Per  Reference  (cb) ,  there  are  eight 
viewpoints  that  are  in  turn  composed  of  more  than  50  various  supporting  products,  subsets  of  which  are 
selected  based  on  program  needs.  The  eight  DoDAF  viewpoints  are  summarized  below: 

•  All  Viewpoint  (AV) .  This  viewpoint  provides  information  pertinent  to  the  entire  Architectural 
Description,  such  as  the  scope  and  context  of  the  Architectural  Description.  These  conditions 
include  doctrine,  TTPs,  relevant  goals  and  vision  statements,  CONOPS,  scenarios,  and 
environmental  conditions. 

•  Capability  Viewpoint  (CV) .  This  viewpoint  captures  the  enterprise  goals  associated  with  the 
overall  vision  for  executing  a  specified  course  of  action.  The  CV  models  are  high  level  and 
describe  capabilities  using  terminology  that  is  easily  understood  by  decision  makers  and  used 
for  communicating  a  strategic  vision  regarding  capability  evolution. 

•  Data  and  Information  Viewpoint  (DIV) .  This  viewpoint  captures  the  business  information 
requirements  and  structural  business  process  rules  for  the  Architectural  Description.  It  describes 
the  information  that  is  associated  with  the  information  exchanges  in  the  Architectural 
Description,  such  as  attributes,  characteristics,  and  interrelationships. 

•  Operational  Viewpoint  (OV) .  This  viewpoint  captures  the  organizations,  tasks,  or  activities 
performed  and  the  information  that  must  be  exchanged  between  them  to  accomplish  DoD 
missions.  It  conveys  the  types  of  information  exchanged,  the  frequency  of  exchange,  which  tasks 
and  activities  are  supported  by  the  information  exchanges,  and  the  nature  of  information 
exchanges. 

•  Project  Viewpoint  (PV) .  This  viewpoint  captures  how  programs  are  grouped  in  organizational 
terms  as  a  coherent  portfolio  of  acquisition  programs.  It  provides  a  way  of  describing  the 
organizational  relationships  between  multiple  acquisition  programs,  each  of  which  is 
responsible  for  delivering  individual  systems  or  capabilities. 

•  Services  Viewpoint  (SvcV) .  This  viewpoint  captures  system,  service,  and  interconnection 
functionality  providing  for,  or  supporting,  operational  activities.  DoD  processes  include 
warfighting,  business,  intelligence,  and  infrastructure  functions.  These  system  functions  and 
service  resources  support  the  operational  activities  and  facilitate  the  exchange  of  information. 

•  Standards  Viewpoint  (StdV) .  This  viewpoint  is  the  minimal  set  of  rules  governing  the 
arrangement,  interaction,  and  interdependence  of  system  parts  or  elements.  The  StdV  provides 
the  technical  systems  implementation  guidelines  upon  which  engineering  specifications  are 
based,  common  building  blocks  are  established,  and  product  lines  are  developed.  It  includes  a 
collection  of  the  technical  standards,  implementation  conventions,  standards  options,  rules,  and 
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criteria  that  can  be  organized  into  profile(s)  that  govern  systems  and  system  or  service  elements 
in  a  given  Architectural  Description. 

•  Systems  Viewpoint  (SV) .  This  viewpoint  captures  the  information  on  supporting  automated 
systems,  interconnectivity,  and  other  systems  functionality  in  support  of  operating  activities  . 

The  DoDAF  StdV,  SV,  SvcV,  DIV,  and  OV  are  of  particular  importance  from  an  interoperability  testing 
perspective. 

Information  exchanges  identified  in  architecture  products  are  first  evaluated  for  standards  conformance 
during  system  development  and  DT,  and  then  later  evaluated  during  OT  for  end-to-end  operational 
effectiveness  and  suitability.  Standards  conformance  testing  is  addressed  in  a  manner  such  that  risks  to 
achieving  full  interoperability  are  mitigated.  Standards  conformance  is  confirmed  during  the  capabilities 
development  process  and  again  through  structured  testing  during  system  development  and  in  DT.  The 
evaluation  of  the  level  of  testing  necessary  should  be  based  on  the  following  factors:  maturity  of  the 
standard,  availability  of  existing  standards  conformance  testing  capabilities,  and  risk  associated  with 
system  conformance  to  a  specified  standard  or  standards  profile. 

Standards  conformance  testing  can  be  costly  and  time-consuming  unless  automated  tools  are  available 
to  assist  in  the  process.  Wherever  possible,  contracts  should  require  that  the  developer  conduct 
standards  conformance  testing  with  adequate  Government  oversight  to  avoid  the  need  to  have  the 
Government  repeat  that  testing  after  the  article  has  been  delivered.  Early  tester  involvement  in  the  RFP 
and  contract  can  help  ensure  that  the  developer  is  responsible  for  conducting  that  testing  and  delivering 
appropriate  documentation.  Options  for  addressing  standards  conformance  are  typically  discussed 
between  the  acquisition  program  office  and  DISA  JITC  to  determine  the  most  cost-effective  method  of 
ensuring  compliance.  Standards  conformance  testing  may  be  based  on  several  factors  include  the 
following: 

•  Accepting  letters  of  compliance  from  vendors  for  mature  standards  (e.g.,  commercial  voice 
switches  or  routers). 

•  Testing  only  military-unique  features  that  are  implemented  over  commercial  standards  (e.g.. 
Defense  Switched  Network). 

•  Testing  entire  MIL-STDs  (e.g.,  tactical  data  links). 

Whenever  possible,  integrated  T&E  should  be  executed  to  maximize  the  reuse  of  interoperability  test 
data.  For  example,  DISA  JITC  assesses  the  sufficiency  of  the  data  submitted  by  the  program  office, 
developer  CDRLs,  and  DT  agent.  The  DT  agent  should  verify  SV-6  requirements  in  advance  of  OT.  OTAs 
should  leverage  data  provided  from  the  PM,  developer,  and  DT  as  entrance  criteria  for  OT.  OT  should  be 
focused  on  validating  the  threshold  requirements  for  the  NR-KPP  as  documented  in  the  critical 
operational  mission  threads  depicted  in  the  OVs. 

In  preparation  for  DT  and  OT,  the  test  agencies  should  collaborate  with  JITC  as  well  as  the  security 
certification  and  accreditation  agent  to  verify  that  the  test  and  security  architecture  is  representative  of 
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the  operational  and  system  views  as  documented  via  the  integrated  architectures  and  security 
documentation  to  identify  any  discrepancies  between  the  test  architecture  and  the  documented  views. 
The  DT  agency  should  verify  MOPs  such  as  timeliness,  accuracy,  and  completeness  of  the  information 
exchanges.  These  data  should  be  shared  with  DISA  JITC  and  the  OTA  to  ensure  that  sufficient  data  are 
collected  to  satisfy  JITC  interoperability  certification  requirements.  Because  of  the  large  number  of 
potential  information  exchanges  during  OT,  evaluations  should  be  scoped  to  select  critical  operational 
activities  based  upon  approved  OVs  and  SVs.  During  the  OT,  the  OTA  evaluates  end-to-end  operational 
effectiveness  of  the  systems  information  exchanges  to  accomplish  the  mission  or  task. 

20.6  Cyber  Security  and  lA  Requirements 

Cyber  security  and  lA  are  an  integral  part  of  the  net-centric  concept  and  T&E.  The  implementation  of 
cyber  security  and  lA  is  defined  by  References  (bu)  and  (bw) .  A  set  of  security  controls  are 
implemented,  which  are  the  safeguards  or  countermeasures  prescribed  for  an  information  system  or 
platform  IT  system  to  protect  the  confidentiality,  integrity,  and  availability  of  the  system  and  its 
information. 

Currently,  minimum  lA  security  controls  are  dependent  upon  the  MAC  and  CL  as  defined  in  Reference 
(bu)  and  as  documented  in  the  lA  acquisition  strategy  and  RMF  package.  lA  controls  that  must  be 
applied  are  defined  in  References  (bt)  and  (bv) . 

References  (bt)  and  (bv)  are  currently  being  revised,  with  publication  expected  to  be  in  FY  2013, 
including  a  definition  of  transition  procedures.  The  revised  policy  will  describe  the  scheme  for 
identifying  the  MAC  and  CL  and  the  lA  security  controls  defined  in  the  NIST  SP  (Reference  (by))  that 
comprehensively  covers  cyber  security  and  lA  and  will  be  used  instead  of  the  DoD  lA  controls  specified 
in  References  (bt)  and  (bv).  DoD-specific  assignment  values,  implementation  guidance,  and  validation 
procedures  for  the  NIST  security  controls  will  be  published  in  the  DoD  Information  Assurance 
Certification  and  Accreditation  Process  (DIACAP)  Knowledge  Service. 

lA  controls  must  be  implemented  and  evaluated  in  conjunction  with  interoperability  to  ensure  that 
those  controls  do  not  adversely  impact  critical  operational  information  exchanges.  Questions  that 
should  be  considered  include  the  following: 

•  Flave  applicable  lA  requirements  of  References  (bt)  and  (bv)  and  Director  of  Central  Intelligence 
Directives  (DCIDs)  and  Intelligence  Community  Directives  (ICDs)  been  identified? 

•  Is  the  system-level  lA  design  (to  include  the  use  of  enterprise  services)  in  alignment  with  the  lA 
component  of  the  GIG  integrated  architecture? 

•  Flas  the  applicable  capability  (system)  received  an  authority  to  operate  (ATO)  from  the 
appropriate  designated  accrediting  authority? 

lA  encompasses  activities  that  protect  and  defend  information  and  systems  by  ensuring  their  availability, 
integrity,  authentication,  confidentiality,  and  non-repudiation.  Compliance  with  References  (bt) ,  (bv) 
and  (bx)  instructions,  regulations,  and  manuals  is  the  responsibility  of  the  materiel  developer  and  the  DT 
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agent  and  is  confirmed  during  the  capabilities  development  process.  Two  aspects  that  are  key  to  lA 
requirements  from  the  tester's  viewpoint  are  formal  certification  and  accreditation  (C&A)  and  the  role 
of  lA  controls. 

The  checklist  for  T&E  assessments  of  interoperability  and  cyber  security  and  the  acquisition  date 
requirements  checklists  for  interoperability  and  cyber  security  are  provided  at  the  end  of  Chapter  20. 

20.6.1  C&A 

In  accordance  with  subchapter  III  of  chapter  35  of  Title  44,  U.S.C.,  all  information  systems  must  be 
certified  and  accredited.  Within  the  DoD,  that  process  is  performed  via  the  DoD  RMF  process. 

RMF  details  will  be  specified  in  the  updates  to  DoDI  8510.01  (Reference  (ca)).  The  RMF  and  C&A  effort  is 
typically  a  long-lead  item  and  is  of  interest  in  TEMP  reviews.  Most  importantly,  systems  must  receive  an 
ATO  or  at  least  an  interim  authority  to  operate  prior  to  conducting  OT  and  an  interim  authority  to  test 
prior  to  conducting  any  lA/network  testing. 

The  RMF  process  replaces  the  DoD  lA  C&A  (DIACAP)  process  and  is  expected  to  be  implemented  in  FY 
2013. 

20.6.2  lA  Controls 

All  sources  of  lA  requirements  must  be  considered  in  testing.  For  all  DoD  systems,  under  current  policy, 
a  set  of  derived  baseline  lA  controls  is  developed  based  on  assessment  of  the  systems  MAC  and  CL  as 
identified  in  the  capabilities  document  and  other  related  documentation  such  as  the  system  security 
authorization  agreement,  SSP,  and  ATO.  The  RMF  process  is  described  at  a  high  level  in  Figure  20-1.  The 
detailed  process  for  assessing  the  system  and  controls  is  described  in  Reference  (ca). 
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■  Conduct  needed  remediation 

-  •  Update  SP  SAR  and  POA&M 
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Figure  20-1  RMF  Process 

Appropriate  lA  controls  (which  may  be  technical,  procedural,  or  a  combination  of  both)  are  integrated 
into  the  system  as  part  of  its  development.  Depending  on  the  MAC  and  CL  requirements,  these  lA 
controls  can  add  substantial  requirements  to  the  system  that  must  be  evaluated  as  part  of  compliance. 

References  (bw)  and  (bx)  can  be  used  by  all  programs  to  shape  lA  evaluations.  The  general  process  is 
generally  depicted  in  Figure  20-2.  The  focus  of  Steps  1  through  4  of  the  DOT&E  process  is  the  allocation 
of  controls  to  the  system  under  test  and  supporting  enclave/computer  network  defense  service  provider 
(CNDSP),  the  verification  of  those  controls  through  RMF  and  DT,  and  the  identification  and  closure  of 
vulnerabilities  in  advance  of  OT&E.  OTAs  execute  Steps  5  and  6  with  a  focus  on  validation  of  lA 
requirements  as  part  of  the  OT.  The  status  and  results  of  each  of  the  RMF  phases  should  be  used  to 
guide  development  and  DT  and  as  exit  criteria  for  DT  and  entrance  criteria  for  OT. 

OTAs  may  conduct  additional  lA  validation  activities  including  red  teaming  and  continuity  of  operations 
(COOP)  evaluation  activities  in  support  of  OT  using  guidelines  provided  by  DOT&E.  Using  such  teams 
during  an  OT,  the  OT  can  demonstrate  the  ability  to  protect,  detect,  respond,  and  restore  systems  in  an 
operational  environment  and  determine  overall  vulnerabilities  with  operations  security  (OPSEC) 
processes  and  procedures. 
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Figure  20-2.  Procedures  for  OT&E  of  lA  in  Acquisition  Programs 

20.6.3  lA  Security  Controls  Testing  Recommendations 

lA  testing  is  a  complex  area  with  many  moving  parts.  Several  studies  have  addressed  ways  to  improve 
the  process.  One  recent  review  of  this  area  recommended  the  following: 

•  Establish  an  Integrated  T&E  IPT  prior  to  RFP/contracting,  which  should  include  interoperability, 
security  C&A  agent,  DT&E,  OT&E,  PM,  and  computer  network  defense  (CND)  representatives. 

•  Address  lA  during  early  system  acquisition  and  engineering  activities  to  ensure  that  lA  controls 
are  being  considered  in  the  design. 

•  Integrated  Test  Teams  (ITTs)  should  collaborate  to  identify  meaningful  and  measurable  lA  test 
criteria  that  address  lA  MOEs  and  MOPs  to  include  end-to-end  lA  MOEs  that  address  system 
operation  in  a  network  context. 

20.6.4  Anti-Tamper  Considerations 

A  topic  sometimes  considered  as  part  of  an  overall  lA  strategy  or  part  of  program  protection 
planning/critical  program  protection  is  anti-tamper  (AT).  For  AT  testing,  the  program  office  develops  an 
AT  validation  plan,  typically  classified,  and  provides  the  necessary  funding  for  the  AT  V&V  on  actual  or 
representative  system  components.  This  plan  is  reviewed  and  approved  by  the  DoD  AT  Executive  Agent. 
The  AT  Executive  Agent  witnesses  these  V&V  activities  and  verifies  that  the  AT  plan  is  implemented  into 
the  system  and  works  according  to  the  AT  plan. 
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20.6.5  Supplier  Risk  Management 

T&E  supports  identification  of  and  countering  risks  related  to  suppliers.  Due  diligence  analysis  for  items 
of  supply  is  performed  to  counter  unintentional  vulnerabilities,  intentional  vulnerabilities  (e.g.,  malicious 
wares  and  processes),  and  counterfeits.  Testing  activities  may  include  software  static  analysis,  dynamic 
analysis  (including  the  use  of  simulation,  white  and  black  box  testing),  penetration  testing,  and  testing  to 
ensure  that  the  component  or  service  is  genuine  (e.g.,  through  tag,  digital  signature,  or  cryptographic 
hash  verification). 

20.6.6  Cyber  Security  and  Operational  Resilience 

Operational  resilience  (being  flexible,  adaptable,  and  successful  in  the  face  of  cyber  degradation,  loss,  or 
attack)  has  three  conditions:  (1)  Information  resources  are  trustworthy,  (2)  Missions  are  prepared  for 
information  resources  degradation  or  loss,  and  (3)  Network  operations  have  the  means  to  prevail  in  the 
face  of  adverse  events.  T&E  supports  the  implementation  of  operational  resilience  by  performing  OT&E 
of  lA  capabilities  to  include  the  ability  to  detect  and  react  to  penetrations  and  exploitations  and  to 
protect  and  restore  data  and  information,  in  order  to  inform  acquisition  and  fielding  decisions.  Testing 
should  include  exercising  under  realistic  cyber  conditions  and  testing  procedures  and  tactics  for  work¬ 
arounds  and  fallbacks  in  the  face  of  hostility.  This  process  may  include  conducting  periodic  exercises  or 
evaluations  of  the  ability  to  operate  during  loss  of  all  information  resources  and  connectivity.  This  also 
includes  being  able  to  dynamically  allocate  information  resources  as  needed  to  sustain  mission 
operations  while  addressing  cyber  failures,  no  matter  the  cause,  and  being  able  to  rapidly  restore 
information  resources  to  a  trusted  state  while  maintaining  support  to  ongoing  missions. 

20.7  Agile  Development  and  T&E 

A  current  trend  in  DoD  acquisition  is  the  use  of  the  agile  development  model,  allowing  acquisition  and 
delivery  of  IT  functionality  in  a  relatively  short  timeframe.  The  agile  development  model  relies  on  a 
closely  knit  development  and  test  team  that  performs  iterative  and  incremental  development,  with 
requirements  and  solutions  evolving  in  a  highly  collaborative  environment.  Implementing  the  agile 
development  model  into  the  standard  DoD  acquisition  life  cycle,  with  existing  documentation  and  test 
requirements  levied  on  a  DoD  major  acquisition  program,  is  a  challenge. 

Agile  integrates  software  testing  into  the  development  process.  Agile-designated  programs  have 
incremental  and  closely  integrated  development  and  testing  cycles,  evolving  functionality  requirements, 
and  intermittent  releases  of  Warfighter-required  functional  components  or  capabilities.  In  this 
environment,  software  testing  in  particular  is  integrated  in  the  development  process  and  formal 
documents,  such  as  the  TEMP,  and  test  events  may  be  significantly  modified  or  replaced  by  other  more 
applicable  documentation  and  events.  These  modified  documentation  requirements  and  events  are 
currently  being  defined. 

Integrated  testing  is  a  key  feature  of  agile  development.  The  agile  model  advocates  creating  an  ITT  to 
facilitate  test  planning,  test  script  development,  test  conduct,  and  data  collection  during  the 
development  and  testing  cycles.  As  the  DoD  joint  interoperability  test  certification  authority,  JITC  is 
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developing  guidance  for  agile-designated  programs  to  support  integrated  testing  in  order  to  evaluate 
joint  interoperability. 

Additional  guidance  on  agile  development  and  agile-designated  programs  is  in  development  and  will  be 
included  or  referenced,  as  appropriate,  when  it  becomes  available. 

20.8  Summary 

The  NR-KPP  components  include  interoperability  and  cyber  security.  These  components  are 
interdependent  and  have  historically  been  tested  and  evaluated  by  stakeholders  throughout  the 
acquisition  process.  The  challenges  for  each  stakeholder  are  to  determine  what  is  essential  and  to 
develop  an  integrated  T&E  methodology  that  tests  essential  elements  and  creates  an  environment  in 
which  data  can  be  shared  by  the  PM,  interoperability,  security  C&A,  DT,  and  OT. 

DISA  JITC,  as  the  DoD  interoperability  certifier,  has  the  primary  responsibility  for  assessing  compliance  of 
the  NR-KPP.  As  with  other  parts  of  a  systems  evaluation,  the  NR-KPP  evaluation  leverages  and  uses  data 
from  DT  and  OT.  lA  is  an  integral  part  of  interoperability  testing.  New  development  methods  such  as 
agile  present  new  challenges  for  interoperability  testing  and  require  close  coordination  with  JITC  and  the 
developer  to  leverage  testing  performed  during  agile  development  for  use  during  interoperability 
certification.  As  the  underlying  components  of  the  NR-KPP  continue  to  evolve,  the  data  required  for 
each  test  will  likewise  evolve.  Close  coordination  between  the  user  community,  materiel  developer, 
security  C&A  agent,  DT  and  OT  test  agencies,  and  JITC  to  ensure  that  the  proper  collected  data  support 
the  data  requirements  of  all  stakeholders  is  crucial  to  fielding  effective,  suitable,  survivable,  and  secure 
systems  to  the  Warfighter. 

CHECKLIST  FOR  T&E  ASSESSMENTS  OF  INTEROPERABILITY 
AND  CYBER  SECURITY 

Objective:  To  ensure  that  interoperability  and  cyber  security  requirements  are  addressed. 

Information  as  appropriate  to  program  development  responsibility 

PM  Responsibilities: 

1.  Latest  program  documentation  (e.g.,  ICD,  CDD,  CPD,  ISP). 

2.  TEMP. 

3.  DT&E  test  plans  and  results/reports. 

Lead  DT  and  OTA  Responsibilities: 

1.  Develop  and  execute  DT&E  test  plans  and  evaluate  results  in  advance  of  OT&E. 

2.  Develop  and  execute  OT&E  test  plan  and  evaluate  results. 


260 


525  &-yR9l3&lzuSya  l-yt-B^SyuDcms 


/KI-L0J^^ 


ACQUISITION  DATA  REQUIREMENTS  CHECKLISTS 
FOR  INTEROPERABILITY  AND  CYBER  SECURITY 

Milestone  A  Checklist 

Objective:  To  ensure  that  interoperability  and  cyber  security/IA  are  addressed  in  requirements 
documents,  concept  refinements,  and  technology  developments. 

Required  Information; 

1.  ICD  that  addresses  the  following: 

a.  lA  requirements. 

b.  CND  architecture. 

c.  Interoperability  requirements. 

d.  Threat  environment. 

2.  T&E  strategy  that  includes  NR  KPP  elements. 

Milestone  B  Checklist 

Objective:  To  ensure  that  interoperability  and  cyber  security/IA  are  addressed  in  requirements 
documents,  concept  refinements,  and  technology  developments. 

Required  Information; 

1.  CDD  and  ISP  that  address  the  following: 

a.  Description  of  operational  environment. 

b.  Intra-  and  inter-platform/systems,  CNDSPs. 

2.  TEMP  with  the  following: 

a.  NR-KPP,  a  list  of  verification  efforts  that  addresses  effectiveness/suitability/  survivability 
of  the  platform,  system,  subsystem,  or  equipment  in  the  intended  operational 
environment. 

b.  Identification  of  lA  testing  roles  and  test  strategy. 

Milestone  C  Checklist 

Objective;  To  identify  the  limitations  and  vulnerabilities  of  the  subject  system,  subsystem,  or  equipment 
and  its  interoperability  and  cyber  security/IA  considerations. 
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Required  Information: 

1.  Latest  program  documentation  (ICD,  CDD,  CPD,  ISP,  performance  specification,  and  SOW). 

2.  TEMP  that  contains: 

a.  System  Description: 

1.  MOEs  and  MOSs  established  for  NR  KPP  and  lA  in  the  CDD  or  CPD. 

2.  Critical  operational  effectiveness  and  suitability  parameters  identified  for  NR 
KPP  and  lA. 

3.  MOEs  and  MOPs  stated  and  evaluation  criteria  and  data  requirements  defined 
to  include  NR  KPP  and  lA. 

b.  System  Introduction: 

1.  Schedule  identifying  NR  KPP  and  lA  events. 

2.  T&E  responsibility  identified  for  NR  KPP  and  lA. 

c.  DT&E  Outline: 

1.  Coordination  with  JITC  identified. 

2.  Description  of  interoperability  test  plans. 

d.  OT&E  Outline: 

1.  Coordination  with  OT&E  identified. 

2.  Identification  of  completion  of  critical  DT  events  as  entrance  criteria  for  OT. 

3.  A  list  of  tests  and  analyses  used  to  determine  the  items  effectiveness/ 
suitability/survivability  in  the  operational  environment. 

3.  Copies  of  the  following  verification  results,  including  T&E  data: 

a.  DT&E  test  plans  and  results/reports. 

b.  Initial  OT&E  test  plans  and  results/reports. 
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Test  &  Evaluation  Management  Guide 
Chapter  21  -  Multi-Service  and  Joint  Testing 

21.1  Introduction 

This  chapter  discusses  the  planning  and  management  of  a  multi-Service  test  program.  A  multi-Service 
test  program  is  conducted  when  a  system  is  to  be  acquired  for  use  by  more  than  one  Service  or  when  a 
system  must  interface  with  equipment  of  another  Service.  A  multi-Service  test  program  should  not  be 
confused  with  an  OSD-sponsored  JT&E  program.  JT&E  is  discussed  section  21.8  of  this  guide. 

21.2  Background 

Multi-Service  T&E  is  conducted  by  two  or  more  DoD  Components  to  address  acquisition  or  interface 
equities  across  each  organization.  All  affected  Services  and  their  respective  OTAs  participate  in  planning, 
conducting,  reporting,  and  evaluating  the  multi-Service  test  program.  One  Service  is  designated  as  the 
lead  Service  and  is  responsible  for  management  of  the  program.  The  lead  Service  is  charged  with  the 
preparation  and  coordination  of  a  single  report  that  reflects  the  systems  operational  effectiveness  and 
suitability  across  all  participating  Services. 

The  management  challenge  in  a  joint  acquisition  program  conducting  multi-Service  T&E  stems  from  the 
fact  that  the  items  undergoing  testing  will  not  necessarily  be  used  by  each  of  the  Services  for  identical 
purposes.  Differences  among  the  Services  usually  exist  in  performance  criteria,  tactics,  doctrine, 
configuration  of  armament  or  electronics,  and  the  operating  environment.  As  a  result,  a  deficiency  or 
discrepancy  considered  disqualifying  by  one  Service  is  not  necessarily  disqualifying  for  all  Services.  It  is 
incumbent  upon  the  lead  Service  to  establish  a  discrepancy  reporting  system  permitting  each 
participating  Service  to  document  all  discrepancies  consistently  within  the  policies  and  criteria  used  by 
each  participating  Service.  At  the  conclusion  of  a  multi-Service  T&E,  each  participating  OTA  may  prepare 
an  lER  in  its  own  format  and  submit  that  report  through  its  normal  Service  channels.  The  lead  Service 
OTA  may  prepare  the  documentation  that  goes  forward  to  the  MDA.  This  documentation  is  coordinated 
with  all  participating  OTAs. 

Formulation  of  a  multi-Service  T&E  program  designates  the  participants  in  the  program  and  gives  a  lead 
Service  responsibility  for  preparing  a  single  report  concerning  a  systems  operational  effectiveness  and 
suitability.  (The  lead  Service  is  the  Service  responsible  for  the  overall  management  of  a  joint  acquisition 
program.  A  supporting  Service  is  a  Service  designated  as  a  participant  in  the  system  acquisition.) 

A  multi-Service  T&E  program  may  include  DT&E  or  OT&E  or  both.  The  OTAs  periodically  execute  a 
formal  MOA  on  MOT&E  that  provides  a  framework  for  the  conduct  of  a  multi-Service  OT  program. 

Figure  21-1  illustrates  the  MOT&E  team  composition. 
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Figure  21-1  MOT&E  Team  Composition 

The  MOA  includes  such  topics  as  duties  and  responsibilities  of  participants,  resourcing  agreements,  DR 
criteria,  and  a  glossary  of  common  terms  and  Service-peculiar  terms.  Resource  requirements  are 
documented  in  the  TEMP.  Generally,  the  process  can  be  summarized  as  follows: 

•  In  a  multi-Service  acquisition  program,  T&E  is  planned  and  conducted  according  to  the  lead 
Service  regulations.  The  designated  lead  Service  will  have  the  overall  responsibility  for 
management  of  the  multi-Service  program  and  will  ensure  that  supporting  Service  requirements 
are  included.  If  another  Service  has  certain  unique  T&E  requirements,  testing  for  these  unique 
requirements  may  be  planned,  funded,  and  conducted  according  to  that  Services  regulations. 

•  Participating  Services  will  prepare  reports  in  accordance  with  their  respective  regulations.  The 
lead  Service  will  prepare  and  coordinate  a  single  DT&E  report  and  a  single  OT&E  report,  which 
will  summarize  the  conclusions  and  recommendations  of  each  Services  reports.  Rationale  will  be 
provided  to  explain  any  significant  differences.  The  individual  Service  reports  may  be  attached 
to  this  single  report. 

•  Deviations  from  the  lead  Service  T&E  regulations  may  be  accommodated  by  mutual  agreement 
among  the  Services  involved 

21.3  Test  Program  Responsibilities 

Each  multi-Service  T&E  program  should  establish  a  WIPT.  Its  membership  consists  of  one  senior 
representative  from  each  participating  Service  or  agency  headquarters.  The  T&E  WIPT  works  closely 
with  the  PMO  and  is  responsible  for  arbitrating  all  disagreements  among  the  Services  that  cannot  be 
resolved  at  the  working  level. 
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Resource  requirements  are  documented  in  the  TEMP.  Each  participating  Service  is  directed  to  budget  for 
the  testing  necessary  to  accomplish  its  assigned  test  objectives  and  for  the  participation  of  its  personnel 
and  equipment  in  the  entire  test  program.  Separate  annexes  may  be  used  to  address  each  Services  test 
requirements.  Funding  will  be  in  accordance  with  public  law  and  Reference  (ao). 

21.4  Test  Team  Structure 

A  sample  test  team  structure,  taken  from  Annex  C  of  the  Memorandum  of  Agreement  (MOA)  on  Multi- 
Service  Operational  Test  and  Evaluation  (MOT&E)  and  Operational  Suitability  Terminology  and 
Definitions  (Reference  (cc)),  is  shown  in  Figure  21-1.  As  shown  in  the  figure,  Service  test  teams  work 
through  a  Service  deputy  test  director  or  senior  representative.  The  lead  Service  OTA  test  director 
exercises  test  management  authority  but  not  operational  control  over  other  Service  test  teams.  The 
responsibilities  include  integration  of  test  requirements  and  efficient  scheduling  of  test  events.  The 
deputy  test  directors  exercise  operational  control  or  test  management  authority  over  their  Service  test 
teams  in  accordance  with  their  Service  directives.  Additionally,  they  act  as  advisers  to  the  test  director; 
represent  their  Services  interests;  and  are  responsible,  at  least  administratively,  for  resources  and 
personnel  provided  by  their  Service. 

21.5  Test  Planning 

Test  planning  for  multi-Service  T&E  is  accomplished  in  the  manner  prescribed  by  lead  Service  directions 
and  overarching  MOA.  The  process  generally  follows  procedures  similar  to  those  outlined  below: 

•  The  lead  Service  T&E  agency  begins  the  planning  process  by  issuing  a  call  to  the  supporting 
Service  T&E  agencies  for  critical  issues  and  test  objectives. 

•  The  lead  Service  T&E  agency  consolidates  the  objectives  into  a  list  and  coordinates  the  list  with 
the  supporting  Service  T&E  agencies. 

•  The  lead  Service  T&E  agency  accommodates  supporting  Service  T&E  requirements  and  input  in 
the  formal  coordination  action  of  the  TEMP. 

•  Participating  T&E  agency  project  officers  assign  responsibility  for  the  accomplishment  of  test 
objectives  (from  the  consolidated  list)  to  each  T&E  agency.  These  assignments  are  made  in  a 
mutually  agreeable  manner.  Each  agency  is  then  responsible  for  resource  identification  and 
accomplishment  of  its  assigned  test  objectives  under  the  direction  of  the  lead  Service  T&E 
agency. 

•  Each  participating  agency  prepares  the  portion  of  the  overall  test  plan(s)  for  its  assigned 
objectives,  in  the  lead  Service  test  plan(s)  format,  and  identifies  its  data  needs. 

•  The  lead  Service  T&E  agency  prepares  the  multi-Service  T&E  test  plan(s),  consolidating  the  input 
from  all  participating  agencies. 
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21.6  Deficiency  Reporting 

In  a  multi-Service  T&E  program,  a  deficiency  report  is  a  report  of  any  condition  that  reflects  adversely  on 
the  item  being  tested  and  that  must  be  reported  outside  the  test  team  for  corrective  action.  The 
deficiency  reporting  system  of  the  lead  Service  is  normally  used.  All  members  of  the  multi-Service  test 
team  will  report  deficiencies  through  their  Services  system. 

Items  undergoing  test  will  not  necessarily  be  used  by  each  of  the  Services  for  identical  purposes.  As  a 
result,  a  deficiency  considered  disqualifying  by  one  Service  is  not  necessarily  disqualifying  for  all 
Services.  Deficiency  reports  of  a  disqualifying  nature  must  include  a  statement  by  the  concerned  Service 
as  to  why  the  deficiency  has  been  so  classified.  The  deficiency  report  also  includes  statements  by  the 
other  Services  as  to  whether  the  deficiency  affects  them  significantly. 

If  one  of  the  participating  Services  identifies  a  deficiency  that  it  considers  as  warranting  termination  of 
the  test,  the  circumstances  are  reported  immediately  to  the  test  director. 

Deficiency  reporting  is  a  key  part  in  any  multi-Service  T&E  MOA.  Figure  21-2  shows  a  sample  deficiency 
report  summary,  as  found  in  Annex  D  of  Reference  (ca). 
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Figure  21-2  Deficiency  Report  Summary  Format 
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21.7  Test  Reporting 

The  following  test-reporting  policy  applies  to  multi-Service  lOT&E: 

•  The  lead  OTA  will  prepare  and  coordinate  the  test  report  reflecting  the  systems  operational 
effectiveness  and  suitability  for  each  Service.  Interim  reports  will  not  normally  be  prepared. 

•  The  test  report  will  synergize  the  different  operational  requirements  and  operational 
environments  of  the  involved  Services. 

•  The  test  report  will  state  findings,  put  those  findings  into  perspective,  and  present  rationale  why 
there  is  or  is  not  consensus  on  the  utility  of  the  system. 

•  All  participating  OTAs  will  sign  the  test  report. 

•  Each  participating  OTA  may  prepare  an  lER  or  final  test  report,  as  required,  in  its  own  format 
and  process  that  report  through  normal  Service  channels. 

•  The  lead  OTA  will  ensure  that  all  separate  participating  Service  lERs/test  reports  are  appended 
to  the  overall  final  test  report  prepared  by  the  lead  OTA  for  submission  to  the  MDA. 

•  The  lead  OTA  will  submit  a  single  integrated  multi-Service  report  to  the  MDA  90  calendar  days 
after  the  official  end  of  test  is  declared  but  not  later  than  45  days  prior  to  a  milestone  decision. 


21.8  JT&E 

JT&E  is  not  the  same  as  multi-Service  T&E.  JT&E  is  a  specific  program  activity  sponsored  by  the  DOT&E. 
JT&E  is  a  means  of  examining  joint-Service  tactics  and  doctrine.  Details  of  formulating  a  JT&E  program 
are  set  forth  in  DoDI  5010.41  (Reference  (cd)).  Figure  21-3  shows  the  formal  process  used  to  identify 
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JT&E  candidates  and  establish  testing  priorities. 


Sponsoring  Services  or 
COCOM  provides 
Government  personnel, 
facilities,and  O&M  support 

DOT&E  provides  dollars  for 
test  specific  support 
including  contractor  support 


Nomination  from 
Services,  COCOMs, 
OSDAeencles 


(Up  to  12  months! 
■Ur  to  $1M) 


Figure  21-3  JT&E  Process 

JT&E  program  objectives  include  the  following:  evaluate  joint  technical  and  operational  concepts  and 
recommend  improvements;  validate  testing  methodologies  that  have  joint  applications;  improve  M&S 
validity  with  field  exercise  data;  increase  joint  mission  capability,  using  quantitative  data  for  analysis; 
provide  feedback  to  the  acquisition  and  joint  operations  communities;  and  improve  joint  TTPs. 

JT&E  is  a  conduit  for  Warfighters  to  solve  joint  operational  issues.  Although  the  DOT&E  manages  the 
program,  partnerships  with  the  Services  and  combatant  commands  are  critical.  JT&E  does  not  assess 
system  performance  and  is  not  a  hardware  acquisition  program. 

The  goal  of  JT&E  is  to  provide  the  following: 


•  Non-materiel  solutions  to  Warfighter  issues. 

•  Process  improvements  so  that  fielded  equipment  is  used  more  effectively  in  a  joint  operational 
environment. 

Test  products  that  are: 
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•  Improved  TTPs,  architectures,  and  training  packages. 

•  New  test  and  evaluation  methodologies 

21.9  Summary 

Multi-Service  weapon  system  acquisition  test  programs  are  conducted  by  two  or  more  Services  when  a 
system  is  to  be  acquired  for  employment  by  more  than  one  Service  or  when  a  system  must  interface 
with  equipment  of  another  Service.  Test  procedures  for  multi-Service  T&E  follow  those  of  the 
designated  lead  Service,  with  mutual  agreements  resolving  areas  in  which  deviations  are  necessary.  The 
MOA  on  OT&E  procedures  provides  guidance  to  the  OTA.  Care  must  be  exercised  when  integrating  test 
results  and  reporting  discrepancies  because  items  undergoing  testing  may  be  used  for  different 
purposes  in  different  Services.  Close  coordination  is  required  to  ensure  that  an  accurate  summary  of  the 
developing  systems  capabilities  is  provided  to  Service  and  DoD  decision  authorities.  Joint  T&E  programs 
involving  interested  Services  are  normally  short-term  TTP-oriented  analytical  efforts  funded  and 
managed  by  the  Office  of  the  DOT&E. 
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Test  &  Evaluation  Management  Guide 
Chapter  22  -  International  T&E  Programs 

22.1  Introduction 

This  chapter  discusses  T&E  from  an  international  perspective.  It  describes  the  OSD-sponsored  FCT 
Program  and  the  international  test  operations  procedures  (ITOPs).  Factors  that  bear  on  the  T&E  of 
multinational  acquisition  programs  are  also  summarized. 

22.2  FCT  Program 

The  Comparative  Technology  Office  (CTO)  is  administered  under  the  Deputy  Assistant  Secretary  of 
Defense  for  Rapid  Fielding  (DASD(RF)).  CTO  has  responsibility  over  the  Defense  Acquisition  Challenge 
(DAC)  Program  and  the  FCT  Program. 

The  purpose  of  the  FCT  Program  is  to  test  items  and  technologies  of  our  allies  and  other  friendly  nations 
that  have  a  high  TRL  in  order  to  satisfy  valid  defense  requirements  more  quickly  and  economically.  Since 
1980,  the  FCT  Program  has  helped  foster  the  two-way  street  in  defense  spending  between  the  United 
States  and  its  allies.  The  purpose  of  the  DAC  Program  is  to  provide  opportunities  for  the  increased 
introduction  of  innovative  and  cost-saving  technology  or  products  into  existing  DoD  acquisition 
programs. 

Each  Military  Department  (Army,  Navy,  and  Air  Force)  and  the  U.S.  Special  Operations  Command 
(USSOCOM)  have  one  or  more  offices  that  manage  their  respective  DAC  and  FCT  programs  and  receive 
funding  and  guidance  from  the  OSD  CTO.  Each  Military  Department  and  USSOCOM  have  acquisition 
authority  for  equipping  their  Warfighters.  Because  the  goal  of  the  DAC  and  FCT  programs  is  to  find  and 
test  the  best  equipment  for  our  Warfighters,  the  Services  and  USSOCOM  are  integral  partners. 

22.2.1  Program  Objectives 

CTO  provides  oversight  direction  to  the  FCT  Program,  the  DAC  Program,  and  other  projects  as  assigned 
by  OSD.  The  FCT  Program  is  designed  to  support  the  evaluation  of  a  foreign  nation's  weapons  system, 
equipment,  or  technology  in  terms  of  its  potential  to  meet  a  valid  requirement  of  one  or  more  of  the 
Military  Services. 

The  CTO  mission  is  to  find,  demonstrate,  transition,  and  rapidly  transfer  the  best  operational  concepts 
and  technology  solutions  for  use  by  the  United  States.  Some  of  these  solutions  may  be  existing  foreign 
systems.  The  FCT  Program  helps  ensure  that  these  systems  are  properly  evaluated  and  that  they  will 
satisfy  U.S.  needs. 

CTO  objectives  are  to  improve  the  U.S.  Warfighters  capabilities  and  reduce  expenditures  through  the 
following: 

•  Rapidly  fielding  quality  military  equipment  12  to  18  months  after  start  (funding)  of  project. 

•  Eliminating  unnecessary  duplication  of  RDT&E. 


270 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcmS 


/KI-L0J!litb 


•  Reducing  life  cycle  or  procurement  costs. 

•  Enhancing  standardization  and  interoperability. 

•  Promoting  competition  by  qualifying  alternative  sources. 

•  Improving  the  U.S.  military  industrial  base. 

•  Promoting  international  technology  exchanges. 

The  program  is  managed  in  accordance  with  section  2350(a)  of  Reference  (b)  and  funded  centrally  by 
the  USD(AT&L). 

22.2.2  Program  Administration 

Foreign  weapons  evaluation  (FWE)  activities  and  responsibilities  are  assigned  to  the  CTO.  Each  year, 
sponsoring  military  Services  nominate  systems  to  be  evaluated  under  the  FCT  Program  to  the  CTO.  The 
Services  are  encouraged  to  make  submissions  whenever  a  promising  candidate  that  appears  to  satisfy  a 
current  or  potential  Service  requirement  is  found.  Procedures  for  nominating  a  system  are  outlined  in 
the  OSD(CTO)  Flandbook  (Reference  (ce)). 

The  fundamental  criterion  for  FCT  Program  selection  is  the  candidate  systems  potential  to  satisfy  U.S. 
operational  or  training  requirements  that  exist  or  are  projected  or  the  systems  possible  contribution  to 
the  U.S.  technology  base.  Additional  factors  influencing  candidate  selection  include  candidate  maturity, 
available  test  data,  multi-Service  interest,  existence  of  a  statement  of  operational  need,  potential  for 
subsequent  procurement,  sponsorship  by  U.S. -based  licensee,  realistic  evaluation  schedule  cost,  DoD 
Component  OSD  evaluation  cost-sharing  proposal,  and  preprogrammed  procurement  funds.  For 
technology  evaluation  programs  within  the  FCT  Program,  the  candidate  nomination  proposal  must 
address  the  specific  arrangements  under  which  the  United  States  and  foreign  participants 
(governments,  armed  forces,  and  corporations)  will  operate.  These  arrangements  may  include 
government-to-government  MOAs,  private  industry  licensing  agreements,  data  exchange  agreements, 
and/or  cooperative  technology  exchange  programs. 

FWE  activities  are  funded  by  OSD  and  executed  by  the  Service  with  the  potential  need  for  the  system. 
Points  of  contact  at  the  FIQ  level  in  each  Service  monitor  the  conduct  of  the  programs.  Work  is 
performed  in  laboratories  and  test  centers  throughout  the  country. 

22.2.3  FCT  Test  Plans 

An  ideal  FCT  T&E  plan  will  use  all  available/acceptable  existing  test  data.  Similarly,  the  plan  should  seek 
to  validate  pass/fail  criteria  in  a  phased  approach  where  possible.  This  approach  reduces  the  DoD 
financial  risk  by  identifying  insurmountable  problems  early  in  the  T&E  process.  Testing  of  items  must  be 
such  as  to  ensure  performance,  operational  effectiveness,  and  operational  suitability  of  an  item  for 
military  application.  A  test  approach  leveraging  previous  test  and  operational  data  of  an  NDI  will  save 
costs. 

OSD  CTO  staff,  as  part  of  the  project  review  process,  will  review  FCT  test  plans,  and  an  approved 
coordinated  test  plan  must  be  in  place  before  commencing  testing. 
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Subpart  225.872  of  the  Defense  Federal  Acquisition  Regulation  Supplement  (Reference  (cf))  waives  the 
Buy  American  Act  for  NATO  and  other  qualifying  countries  when  using  the  FCT  Program. 

22.3  NATO  Comparative  Test  (NCT)  Program 

The  NCT  Program  has  been  integrated  with  the  FCT  Program.  The  NCT  Program  was  created  per 
Reference  (az).  The  program  supports  the  evaluation  of  NATO  nations'  weapons  systems,  equipment, 
and  technology  and  assesses  their  suitability  for  use  by  U.S.  forces.  NCT  program  selection  criteria  are 
essentially  the  same  as  for  the  FCT  Program.  The  exceptions  are  that  the  equipment  must  be  NATO 
member-nation  produced  and  be  considered  as  an  alternative  to  a  system  in  a  late  stage  of 
development  in  the  U.S.  or  offer  a  cost,  schedule,  or  performance  advantage  over  U.S.  equipment.  In 
addition,  the  NCT  Program  must  notify  the  Armed  Services  and  Appropriations  Committees  of  the  Flouse 
of  Representatives  and  Senate  before  funds  are  obligated.  With  this  exception,  the  NCT  Program  follows 
the  same  nomination  process  and  administrative  procedures  as  the  FCT  Program. 

Problems  associated  with  testing  foreign  weapons  normally  stem  from  politics,  national  pride,  and  a  lack 
of  previous  test  data.  The  test  community  must  be  careful  not  to  allow  national  pride  to  be  a  driving 
force  in  the  evaluation.  U.S.  testers  must  make  every  effort  to  obtain  all  available  test  data  on  foreign 
systems.  These  data  can  be  used  to  help  validate  the  evolving  test  data  and  additional  test  data  during 
the  evaluation. 

22.4  T&E  Management  in  Multinational  Programs 
22.4.1  Compatibility  With  Allies 

Rationalization,  standardization,  and  interoperability  have  become  increasingly  important  elements  in 
the  materiel  acquisition  process.  CJCSI  2700. OlE  (Reference  (eg))  requires  that  equipment  for  use  by 
personnel  of  the  U.S.  Armed  Forces  stationed  in  Europe  under  the  terms  of  the  North  Atlantic  T reaty  be 
standardized  or  at  least  interoperable  with  equipment  of  other  NATO  members.  Therefore,  PMs  and 
TMs  must  be  fully  aware  of  any  potential  international  applications  of  the  systems  for  which  they  are 
responsible. 

Representatives  of  the  United  States  and  other  NATO  countries  have  executed  MOAs  concerning  the 
mutual  acceptability  of  each  country's  T&E  data.  Such  MOAs  seek  to  avoid  redundant  testing  by 
documenting  the  extent  of  understanding  among  involved  governments  concerning  mutual  acceptability 
of  respective  T&E  procedures  for  systems  that  are  developed  in  one  country  and  are  candidates  for 
procurement  by  one  or  more  of  the  other  countries.  Focal  points  for  DT  and  OT  in  each  of  the  countries 
are  identified,  and  procedures  governing  generation  and  release  of  T&E  data  are  described  in  the  MOU. 
The  European  concept  of  OT&E  is  significantly  different  from  that  used  by  the  U.S.  DoD. 

Early  and  thorough  planning  is  an  important  element  of  any  successful  T&E  program  but  is  even  more 
critical  in  a  multinational  program.  Agreement  must  be  reached  concerning  T&E  procedures,  data 
requirements,  and  methodology.  Differences  in  tactics,  battlefield  representations,  and  military 
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organizations  may  make  it  difficult  for  one  nation  to  accept  another's  test  data.  Therefore,  agreement 
must  be  reached  in  advance  concerning  the  OT  scenario  and  battlefield  representation  that  will  be  used. 

22.4.2  ITOPs 

ITOPs  are  documents  containing  standardized  state-of-the-art  test  procedures  prepared  by  the 
cooperative  efforts  of  France,  Germany,  the  United  Kingdom,  and  the  United  States.  Their  use  ensures 
high  quality,  efficient,  accurate,  and  cost-effective  testing.  Per  Reference  (c)  the  DOT&E  is  the  OSD 
sponsor  for  providing  basic  guidance  and  direction  to  the  ITOP  processes.  The  ITOPs  program  is  intended 
to  shorten  and  reduce  costs  of  the  materiel  development  and  acquisition  cycle,  minimize  duplicate 
testing,  improve  interoperability  of  U.S.  and  allied  equipment,  promote  the  cooperative  development 
and  exchange  of  advanced  test  technology,  and  expand  the  customer  base.  Each  Service  has  designated 
an  ITOPs  point  of  contact.  The  Army  is  the  lead  Service.  Completed  documents  are  submitted  to  DTIC  for 
official  distribution. 

22.5  U.S.  and  NATO  Acquisition  Programs 

Some  test  programs  involve  combined  development  and  test  of  new  weapon  systems  for  U.S.  and  other 
NATO  countries.  In  these  programs,  some  differences  from  the  regular  normal  methodologies  occur.  An 
international  agreement  is  concluded  with  one  or  more  foreign  governments  including  their  agencies, 
instrumentalities,  or  political  subdivisions  or  with  an  international  organization.  DoDD  5530.3  (Reference 
(ch)),  is  the  principal  directive  that  governs  international  agreements.  However,  for  cooperative  R&D 
international  agreements.  References  and  (])  and  the  Office  of  the  USD(AT&L)  International 
Armaments  Cooperation  Handbook  (Reference  (ci)),  delineate  streamlined  procedures  that  may  be  used 
in  lieu  of  the  lengthy  requirements  mandated  by  Reference  (ch) .  The  definition  of  an  international 
agreement  contains  important  aspects.  It  can  be  concluded  by  any  DoD  Component,  or  in  certain 
situations  by  the  Department  of  State,  with  a  foreign  government  or  international  organization.  The 
United  States  insists  that  any  international  agreement  must  signify  the  intention  of  its  parties  to  be 
bound  in  international  law.  Although  Reference  (ch)  lists  many  possible  denominations  for  an 
international  agreement,  the  most  common  are  MOUs  or  MOAs.  Usually,  a  multinational  MOU  or  MOA 
is  created  concerning  test  program  and  production  funding,  test  resources,  test  team  composition,  use 
of  national  assets  for  testing,  etc.  Nations  are  encouraged  to  use  the  data  that  another  nation  has 
gathered  on  similar  test  programs  to  avoid  duplication  of  effort. 

22.6  Summary 

The  procurement  of  weapon  systems  from  foreign  nations  for  use  by  U.S.  Armed  Forces  can  provide  the 
following  advantages:  reduced  R&D  costs,  faster  IOC,  improved  interoperability  with  friendly  nations, 
and  lower  procurement  costs  because  of  economies  of  scale.  This  procurement  is  normally  a  preferred 
solution  to  user  requirements  before  attempting  to  start  a  new  development.  Testing  such  systems 
presents  specific  challenges  to  accommodate  the  needs  of  all  users.  OT&E  conducted  by  allied  and 
friendly  nations  may  not  follow  U.S.  public  law  and  DoD  acquisition  regulations.  Such  testing  requires 
careful  advance  planning  and  systematic  execution.  Expectations  and  understandings  must  be  well 
documented  at  an  early  stage  to  ensure  that  the  test  results  have  utility  for  all  concerned. 
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Test  &  Evaluation  Management  Guide 

Chapter  23  -  Commercial  Off-The-Shelf  and  Non-Developmental  Items 
Considerations 

23.1  Introduction 

Many  options  are  available  when  an  acquisition  strategy  calls  for  a  materiel  solution  before  electing  to 
develop  a  new  system.  They  range  from  the  preferred  solution  of  using  commercially  available  products 
from  domestic  or  international  sources  to  ultimately  the  least  preferred  option  of  a  traditional,  single- 
Service  new  development  program.  Between  these  two  extremes  are  other  acquisition  strategies  that 
can  call  for  using  modified  systems  at  different  component  levels  and  unmodified  or  ruggedized 
cooperative  developments  to  various  extents.  Figure  23-1  illustrates  the  spectrum  of  approaches  that 
can  be  taken.  Complex  DoD  systems  typically  use  a  variety  of  such  approaches.  No  matter  what 
combination  of  approaches  is  used,  T&E  must  be  performed.  This  chapter  discusses  some  key  COTS  and 
NDI  testing  considerations.  Additionally,  software  COTS  testing  considerations  are  discussed  in  Chapter 
15  of  this  guide. 
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Figure  23-1  Spectrum  of  COTS/NDI  Applicability  (not  to  scale) 


23.1.1  Definitions 


Section  2.101  of  Reference  (al)  formally  defines  Cl.  Generally,  a  Cl  is  defined  as  follows: 
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Any  item,  other  than  real  property,  that  is  of  a  type  customarily  used  for  nongovernmental  purposes 
and: 


•  Has  been  sold,  leased,  or  licensed  to  the  general  public;  or 

•  Has  been  offered  for  sale,  lease,  or  license  to  the  general  public. 

Any  item  that  evolved  through  advances  in  technology  or  performance  and  that  is  not  yet  available  in 
the  commercial  marketplace  but  will  be  available  in  the  commercial  marketplace  in  time  to  satisfy  the 
delivery  requirements  under  a  Government  solicitation. 

Also  included  in  section  2.101  of  Reference  (al)  are  services  in  support  of  a  Cl  or  services  of  a  type 
offered  and  sold  competitively  in  substantial  quantities  in  the  commercial  marketplace  based  on 
established  catalog  or  market  prices  for  specific  tasks  performed  under  standard  commercial  terms  and 
conditions;  this  does  not  include  services  that  are  sold  based  on  hourly  rates  without  an  established 
catalog  or  market  price  for  a  specific  service  performed. 

Per  section  2.101  of  Reference  (al)  an  NDI  is  defined  as  follows: 

(1)  Any  previously  developed  item  of  supply  used  exclusively  for  governmental  purposes  by  a  Federal 
agency,  a  State  or  local  government,  or  a  foreign  government  with  which  the  United  States  has  a  mutual 
defense  cooperation  agreement; 

(2)  Any  item  described  in  paragraph  (1)  of  this  definition  that  requires  only  minor  modification  or 
modifications  of  a  type  customarily  available  in  the  commercial  marketplace  in  order  to  meet  the 
requirements  of  the  procuring  department  or  agency;  or 

(3)  Any  item  described  in  paragraphs  (1)  or  (2)  solely  because  the  item  is  not  yet  in  use. 

All  NDI  systems  are  required  to  undergo  technical  and  OT&E  before  the  procurement  decision,  unless 
the  decision  authority  determines  that  previous  testing  or  other  data  (such  as  user/market 
investigations)  provide  sufficient  evidence  of  acceptability. 

23.1.2  Advantages  and  Disadvantages  of  Commercial  and  NDI  Approaches 

The  advantages  of  using  CIs  and  NDIs  include  the  following: 

•  The  time  to  field  a  system  can  be  greatly  reduced,  providing  a  quicker  response  to  the  user's 
needs. 

•  R&D  costs  or  TOCs  may  be  reduced. 

•  State-of-the-art  technology  may  be  available  sooner. 

The  disadvantages  of  using  CIs  and  NDIs  include  the  following: 

•  Commercial  and  NDI  acquisitions  can  create  logistics  support  difficulties. 

•  Commercial  and  NDI  acquisitions  may  not  have  comparable  competition;  therefore,  fewer 
second  sources  may  be  available. 
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•  With  commercial  and  NDI  acquisitions,  engineering  and  test  data  often  are  not  available. 

•  With  commercial  and  NDI  acquisitions,  it  is  often  difficult  to  reach  agreement  among  the 
product  provider,  developer,  and  user  on  the  level  of  modification  required  for  military  utility. 

•  Commercial  and  NDI  acquisitions  data  rights  and  software  updates. 

23.1.3  COTS  and  NDI  Operating  Environments 

Ideally,  CIs  or  NDIs  will  be  used  in  the  same  environment  and  operating  conditions  and  with  the  same 
user  skill  base  for  which  the  items  were  originally  designed.  Such  items  normally  do  not  require  DT  prior 
to  PQT  except  in  those  cases  in  which  a  contract  may  be  awarded  to  a  contractor  that  has  not  previously 
produced  an  acceptable  finished  product  and  the  item  is  assessed  as  high  risk.  In  that  case, 
preproduction  qualification  testing  would  be  required.  Also,  an  OA  or  some  more  rigorous  level  of  OT&E 
might  be  appropriate. 

In  the  DoD,  however,  COTS  or  NDIs  can  be  used  in  an  environment  other  than  that  for  which  the  items 
were  originally  designed.  This  is  a  potential  (and  frequently  overlooked)  significant  risk  area.  Such  items 
may  require  DoD-unique  modifications  in  their  hardware  and/or  software  that  add  complexity  to 
development,  support,  and  T&E.  Such  items  require  testing  in  an  operational  environment, 
preproduction  qualification  testing  (if  previous  testing  or  modifications  for  DoD  needs  resulted  in  item 
redesign),  and  PQT. 

Integration  of  CIs  or  NDIs  into  a  newly  developed  system  may  require  some  degree  of  regression  testing. 
These  efforts  require  more  extensive  RDT&E  achieve  effective  operation  of  the  desired  system 
configuration,  which  is  frequently  complicated  by  the  lack  of  availability  of  COTS/NDI  engineering  and 
test  data  or  by  proprietary  considerations  relating  to  release  of  such  data.  Required  testing  includes 
feasibility  testing  in  a  military  environment,  preproduction  qualification  testing,  hardware/software 
integration  testing,  OT,  and  PQT.  Additional  certification  testing  may  be  required,  such  as 
transportability  (shipboard  and  airlift)  or  lA  testing. 

Given  the  variety  of  approaches  that  may  be  employed,  it  is  imperative  that  the  acquisition  strategy 
clearly  specify,  with  the  agreement  of  the  testing  authority,  the  following: 

•  The  level  of  testing  that  will  be  performed  on  CIs  and  NDI  systems. 

•  Access  to  requisite  technical  data. 

•  Understanding  how  the  CQTS/NDI  environment  differs  from  the  envisioned  DoD  usage. 

23.2  Market  Investigation  and  Procurement 

A  market  investigation  is  the  central  activity  leading  to  the  program  initiation  decision  regarding  the  use 
of  a  Cl  or  NDI  acquisition  strategy.  The  purpose  of  the  market  investigation  is  to  determine  the  nature  of 
available  products  and  the  number  of  potential  vendors.  Market  investigations  may  vary  from  informal 
telephone  inquiries  to  comprehensive  industry-wide  reviews.  During  the  market  investigation,  sufficient 
data  must  be  gathered  to  support  a  definitive  decision,  to  finalize  the  requirements,  and  to  develop  an 
acquisition  strategy  that  is  responsive  to  these  requirements.  The  search  would  include  dual-use 
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technologies  that  meet  military  needs,  yet  have  sufficient  commercial  application  to  support  a  viable 
production  base. 

During  the  market  investigation,  a  formal  request  for  information  process  may  be  followed  wherein  a 
brief  narrative  description  of  the  requirement  is  published  and  interested  vendors  are  invited  to 
respond.  Test  samples  or  test  items  may  be  leased  or  purchased  at  this  time  to  support  the  conduct  of 
operational  suitability  tests,  to  evaluate  the  ability  of  the  equipment  to  satisfy  the  requirements,  and  to 
help  build  the  functional  purchase  description  or  system  specification.  This  type  of  preliminary  testing 
should  not  be  used  to  select  or  eliminate  any  particular  vendor  or  product  unless  it  is  preceded  by 
competitive  contracting  procedures. 

It  is  imperative  that  technical  and  operational  evaluators  become  involved  during  this  early  stage  of  any 
Cl  or  NDI  procurement  and  that  they  perform  an  early  assessment  of  the  initial  issues.  The  evaluators 
must  also  relate  these  issues  to  T&E  criteria  and  provide  their  independent  evaluation  plans  and  reports 
to  the  decision  authorities  in  a  timely  manner. 

23.3  COTS  and  NDI  Testing 

23.3.1  General  Considerations 

T&E  must  be  considered  throughout  the  acquisition  of  a  system  that  involves  CIs  and  NDIs.  Test  planning 
for  COTS  and  NDIs  should  recognize  prior  commercial  experience  and  testing  performed  on  the 
products.  Nonetheless,  the  appropriate  levels  of  DT&E,  OT&E,  and  LFT&E  are  needed  to  ensure  effective 
performance  in  the  intended  operational  environment.  The  program  office  should  develop  an 
appropriate  T&E  strategy  for  CIs.  This  strategy  should  include  evaluating  CIs  in  a  system  test  bed,  when 
appropriate;  focusing  such  test  beds  on  high-risk  items;  and  testing  Cl  upgrades  for  unanticipated  side 
effects  in  areas  such  as  security,  safety,  reliability,  and  performance. 

The  amount  and  level  of  testing  required  depends  on  the  nature  of  the  Cl  or  NDI  and  its  anticipated  use; 
testing  should  be  planned  to  support  the  design  and  decision  process.  At  a  minimum,  T&E  will  be 
conducted  to  verify  integration  and  interoperability  with  other  system  elements.  All  Cl  and  NDI 
modifications  necessary  to  adapt  the  items  to  the  weapon  system  environment  will  also  be  subject  to 
T&E.  Available  test  results  from  all  commercial  and  Government  sources  may  help  determine  the  actual 
extent  of  testing  necessary.  In  medical  programs,  preliminary  test  results  may  be  used  for  down- 
selecting  items  for  further  T&E. 

For  example,  a  Cl  or  NDI  usually  encompasses  a  mature  design.  The  availability  of  this  mature  design 
contributes  to  the  rapid  development  of  the  logistics  support  system  that  will  be  needed.  In  addition, 
there  are  more  production  items  available  for  use  in  a  test  program.  These  systems  also  require  activity 
in  areas  associated  with  traditional  development  and  acquisition  programs.  For  example,  training  and 
maintenance  programs  and  manuals  must  be  developed,  and  sufficient  time  should  be  allowed  for  their 
preparation. 
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When  the  solicitation  package  for  a  Cl  or  NDI  acquisition  is  assembled,  it  should  include  the  following 
T&E-related  items: 

•  Approved  T&E  issues  and  criteria. 

•  A  requirement  that  the  contractor  provide  a  description  of  the  testing  previously  performed  on 
the  system,  including  test  procedures  followed,  data,  and  results  achieved. 

•  PQT  and  quality  conformance  requirements. 

•  Acceptance  test  plans  for  the  system  and  its  components. 

23.3.2  Testing  Before  Program  Initiation 

Because  reduced  acquisition  time  is  an  important  advantage  of  using  a  Cl  or  NDI  acquisition  strategy,  it 
is  important  that  testing  not  be  redundant  and  that  it  be  limited  to  the  minimum  effort  necessary  to 
obtain  the  required  data.  Testing  can  be  minimized  by: 

•  Obtaining  and  assessing  contractor  test  results. 

•  Obtaining  usage/failure  data  from  other  customers. 

•  Observing  contractor  testing. 

•  Obtaining  T&E  results  from  independent  test  organizations  (e.g..  Underwriters  Laboratories.) 

•  Verifying  selected  contractor  test  data. 

If  it  is  determined  that  more  information  is  needed  after  the  initial  data  collection  from  the  above 
activities,  CIs  or  NDI  candidates  may  be  bought  or  leased,  and  technical  and  operational  tests  may  be 
conducted. 

23.3.3  Testing  After  Program  Initiation 

All  testing  conducted  after  the  program  initiation  milestone  decision  to  proceed  with  the  Cl  or  NDI 
acquisition  solution  should  be  described  in  the  Acquisition  Strategy  and  the  TEMP.  DT  is  conducted  only 
if  specific  information  that  cannot  be  satisfied  by  contractor  or  other  test  data  sources  is  needed.  OT  is 
conducted  as  needed,  based  on  the  risk  assessment  provided  by  the  test  agency. 

T&E  continues  even  after  the  system  has  been  fielded.  This  testing  takes  the  form  of  a  follow-on 
evaluation  to  validate  and  refine  operating  and  support  cost  data;  RAM  characteristics;  logistics  support 
plans;  and  training  requirements,  doctrine,  and  tactics. 

23.3.4  Special  Considerations 

Medical  programs,  by  their  nature  of  almost  exclusively  consisting  of  GOTS  and  COTS  items  and  NDIs, 
and  with  the  inclusion  of  other  Government  agencies  participation  (i.e.,  the  Food  and  Drug 
Administration  (FDA)),  have  special  circumstances  surrounding  their  testing. 

Medical  materiel  is  governed  as  a  program  with  significant  oversight  by  the  PM  and  with  performance- 
based  requirements  composed  by  an  IPT  or  a  high-performance  team  (FIPT).  The  Defense  Medical 
Materiel  Program  Office  (DMMPO),  PMs,  joint  and  Service  procurement  agencies.  Service  T&E  activities. 
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and  other  governmental  organizations  assist  in  the  development  of  performance  criteria  for  T&E,  for 
both  developmental  and  non-developmental  medical  materiel,  as  stipulated  in  DoDI  6430.02  (Reference 
(cj)).  Efficacy  is  based  on  the  products  clearance  for  use  by  the  FDA,  if  applicable,  and  compliance  with 
the  FDAs  rules  governing  manufacture  (i.e..  Good  Manufacturing  Practices).  The  FDA  classifies  medical 
devices  into  three  categories,  each  more  rigorously  examined  than  the  last,  but  no  testing  is  performed 
by  the  FDA  on  these  items. 

DT  of  military  medical  devices  is  normally  limited  to  airworthiness  certification  and  environmental 
testing  to  ensure  that  the  device  does  not  fail  due  to  the  austere  or  harsh  environments  imposed  by  the 
operational  environment  or  interfere  with  the  aircraft  where  it  may  be  utilized.  This  testing  can  often  be 
integrated  into  or  performed  alongside  the  requisite  OT.  Like  items  of  medical  devices  and  equipment 
can  be  difficult  to  distinguish  in  terms  of  overall  construction  or  capability;  therefore,  effectiveness  and 
suitability  (aspects  of  OT)  become  the  primary  delineating  factors  for  selection.  This  performance  under 
pressure,  whether  environmental  or  the  stresses  of  the  user,  distinguishes  the  medical  devices  that  will 
fit  the  unique  needs  of  military  operations  from  those  that  are  better  used  in  controlled  hospital 
conditions. 

23.4  Resources  and  Funding 

Programming  and  budgeting  for  a  Cl  or  NDI  acquisition  present  a  special  challenge.  Because  of  the 
shorter  duration  of  the  acquisition  process,  the  standard  lead  times  required  in  the  normal  budgeting 
cycle  may  be  unduly  restrictive.  This  situation  can  be  minimized  through  careful,  advance  planning  and, 
in  the  case  of  urgent  requirements,  reprogramming/supplemental  funding  techniques. 

RDT&E  funds  are  normally  used  to  support  the  conduct  of  the  MSA  phase  and  the  purchase  or  lease  of 
candidate  systems/components  required  for  T&E  purposes.  The  RDT&E  funds  are  also  used  to  support 
T&E  activities  such  as  modification  of  the  test  article;  purchase  of  specifications,  manufacturers 
publications,  repair  parts,  and  special  tools  and  equipment;  transportation  of  the  test  article  to  and  from 
the  test  site;  and  training,  salaries,  and  temporary  duty  costs  of  T&E  personnel.  Procurement, 
operations,  and  maintenance  funds  are  usually  used  to  support  P&D  costs. 

One  reason  for  using  a  Cl  or  NDI  acquisition  strategy  is  the  potential  for  reduced  overall  TOC.  Additional 
cost  savings  can  be  achieved  after  a  contract  has  been  awarded  if  the  PM  ensures  that  incentives  are 
provided  to  contractors  to  submit  value  engineering  change  proposals  to  the  Government  when 
unnecessary  costs  are  identified. 

23.5  Summary 

The  use  of  CIs  and  NDIs  in  a  system  acquisition  can  provide  considerable  time  and  cost  savings.  The 
testing  approach  used  must  be  carefully  tailored  to  the  type  of  system,  levels  of  modifications, 
technology  risk  areas,  and  the  amount  of  test  data  already  available.  The  T&E  community  must  get 
involved  early  in  the  process  so  that  all  test  issues  are  adequately  addressed  and  timely  comprehensive 
evaluations  are  provided  to  decision  authorities. 
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Test  &  Evaluation  Management  Guide 
Chapter  24  -  Testing  the  Special  Cases 

24.1  Introduction 

This  chapter  covers  the  special  factors  and  alternative  test  strategies  that  the  tester  should  consider  in 
testing  dangerous  or  lethal  weapons,  one-of-a-kind  or  limited  production  systems,  joint  capability 
technology  demonstrations,  and  systems  with  high-cost  and/or  special  security  considerations. 
Examples  include  chemical,  biological,  and  medical  (CBM)  defensive  systems;  laser  weapons;  and  space 
and  missile  systems. 

24.2  Testing  with  Limitations 

Certain  types  of  systems  cannot  be  tested  using  relatively  standard  T&E  for  reasons  such  as  a 
nonstandard  acquisition  strategy,  resource  limitations,  cost,  safety,  environmental  conditions,  or 
security  constraints.  In  such  cases,  the  TEMP  should  contain  a  statement  that  identifies  those  factors 
that  will  preclude  a  full  and  completely  realistic  OT  (lOT&E  and  FOT&E).  The  impact  of  these  limitations 
on  the  tests  COIs  must  also  be  addressed. 

Nonstandard  acquisition  strategies  are  often  used  for  one-of-a-kind  or  limited  production  systems. 
Examples  of  these  include  space  systems,  missiles,  and  ships.  For  one-of-a-kind  systems,  the  production 
decision  is  often  made  prior  to  system  design;  therefore,  testing  does  not  support  the  traditional 
decision  process.  In  limited  production  systems,  there  are  often  no  prototypes  available  for  test; 
consequently,  the  tester  must  develop  innovative  test  strategies. 

Medical  programs,  by  their  nature  of  almost  exclusively  consisting  of  GOTS  and  COTS  items  and  NDIs, 
and  with  the  inclusion  of  other  Government  agencies  participation  (i.e.,  the  FDA),  have  special 
circumstances  surrounding  their  testing.  Medical  materiel  are  governed  as  programs  with  significant 
oversight  by  the  PM  and  with  performance-based  requirements  composed  by  an  IPT  or  an  FIPT.  The 
DMMPO,  PMs,  joint  and  Service  procurement  agencies.  Service  T&E  activities,  and  other  governmental 
organizations  assist  in  the  development  of  OT  and  performance  evaluation  criteria  for  T&E,  for  both 
developmental  and  nondevelopmental  medical  materiel  (As  stipulated  in  Reference  (cp) .  Efficacy  has 
already  been  shown,  based  on  the  products  clearance  for  use  by  the  FDA,  if  applicable,  and  compliance 
with  the  FDAs  rules  governing  manufacture  (i.e..  Good  Manufacturing  Practices).  The  FDA  classifies 
medical  devices  into  three  categories,  each  more  rigorously  examined  than  the  last,  but  no  testing  is 
performed  by  the  FDA  on  these  items. 

Testing  of  military  medical  devices,  because  of  the  reliance  on  COTS  items  cleared  for  use  by  the  FDA, 
does  not  typically  involve  the  rigorous  DT  imposed  on  other  systems.  Unless  the  item  is  developed 
specifically  for  military  use,  DT  is  normally  limited  to  airworthiness  certification  and  environmental 
testing  to  ensure  that  the  device  does  not  fail  due  to  the  austere  or  harsh  environments  imposed  by  the 
operational  scenario.  This  testing  can  often  be  integrated  into  or  performed  alongside  the  requisite  OT. 
Like  items  of  medical  materiel  can  be  difficult  to  distinguish  in  terms  of  overall  construction  or 
capability;  therefore,  effectiveness  and  suitability  (aspects  of  OT)  become  the  primary  delineating 


280 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcmS 


/KI-L0J!liK 


factors  for  selection.  This  performance  under  pressure,  whether  environmental  or  the  stresses  of  the 
user,  distinguishes  the  medical  devices  that  will  fit  the  unique  needs  of  military  operations  from  those 
that  are  better  used  in  controlled  hospital  conditions. 

Medical  devices  must  also  comply  with  the  SEP  in  terms  of  Public  Law  104-191  (Reference  (ck))  and 
DIACAP  information  protection  procedures  and  measures.  Most  medical  devices  will  require  Information 
Management/IT  testing  and  validation  of  information  security  protocols.  The  Army  and  Air  Force  have 
dedicated  medical  OT  agencies,  separate  from  their  Service  OTAs.  These  medical  OT  agencies  are 
empowered  by  their  respective  OTAs  to  conduct  OT  for  medical  materiel,  and  both  have  sites  in  place  to 
accommodate  Information  Management/IT  testing.  Depending  on  structure,  these  responsible  medical 
test  agencies  work  with  the  PM  and  the  IPT/HPT  to  evaluate  risk,  address  COIs,  and  conduct  OT  events. 
Each  medical  test  agency  evaluates  what  type  of  testing  will  be  required  and  how  best  to  mitigate  risks 
incumbent  to  fielding  the  system  or  device.  Service  guidelines  provide  additional  information  about 
these  structures  and  processes. 

The  tester  of  dangerous  or  lethal  systems,  like  chemical  and  laser  weapons,  must  consider  various 
safety,  health,  and  medical  factors  in  developing  test  plans,  such  as: 

•  Provision  of  medical  facilities  for  pre-  and  post-test  checkups  and  emergency  treatment. 

•  Need  for  protective  gear  for  participating/observer  personnel. 

•  Approval  of  the  test  plan  by  the  Surgeon  General. 

•  Restrictions  in  selection  of  test  participants  (e.g.,  medical  criteria,  use  of  only  volunteers). 

•  Restricted  test  locations. 

•  Safety  approvals  involving  range  needs  such  as  flight  termination  systems  and  large  hazard 
footprints. 

•  EISs  and  compliance  with  ESOH  legal  and  regulatory  requirements. 

Also,  the  tester  must  allow  for  additional  planning  time,  test  funds,  and  test  resources  to  accommodate 
such  factors. 

24.2.1  CBM  Testing 

The  testing  of  chemical  and  biological  defense  equipment  and  medical  countermeasures  systems  (such 
as  chemical  and  biological  detection  and  reconnaissance  systems,  individual  and  collective  protection 
systems,  decontamination  systems,  information  management  systems,  medical  devices,  drugs  and 
vaccines,  and  installation  and  force  protection  systems)  poses  unique  problems  because  the  tester 
cannot  perform  actual  open-air  field  testing.  In  addition  to  obvious  health  and  safety  factors,  test  issues 
that  the  CBM  system  tester  must  address  include  the  following: 

•  All  possible  chemical  reactions  due  to  variations  such  as  moisture,  temperature,  pressure,  and 
contamination. 

•  Physical  behavior  of  the  chemical;  i.e.,  droplet  size,  dispersion  density,  and  ground 
contamination  pattern  when  used  operationally. 

•  Toxicity  of  the  chemical;  i.e.,  lethality  and  duration  of  contamination  when  used  operationally. 

•  Safety  of  the  chemical  weapon  during  storage,  handling,  and  delivery. 
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•  Decontamination  processes. 

Addressing  all  of  these  issues  requires  a  combination  of  laboratory  chamber  tests  and  open-air  field 
testing.  The  latter  must  be  performed  using  simulants,  which  are  substances  that  replicate  the  physical 
and  chemical  properties  of  the  tested  agent  but  with  no  toxicity. 

The  development  and  use  of  simulants  for  testing  will  require  increased  attention  as  more  chemical 
weapons  are  developed.  Chemical  and  biological  agents  can  demonstrate  a  wide  variety  of  effects 
depending  on  such  factors  as  moisture,  temperature,  and  contamination.  Consequently,  the  simulants 
must  be  able  to  replicate  all  possible  agent  reactions;  it  is  likely  that  several  simulants  would  have  to  be 
used  in  a  test  to  produce  all  predicted  agent  behaviors.  In  developing  and  selecting  simulants,  the  tester 
must  thoroughly  understand  all  chemical  and  physical  properties  and  possible  reactions  of  the  agent. 

Studies  of  the  anticipated  reactions  can  be  performed  in  toxic-chamber  tests  using  the  real  agent.  Here, 
factors  such  as  changes  in  moisture,  temperature,  pressure,  and  levels  of  impurity  can  be  controlled  to 
assess  the  agent's  behavior.  The  tester  must  think  through  all  possible  environmental  conditions  in 
which  the  system  could  operate  so  all  cases  can  be  tested  in  the  laboratory  chamber  with  the  real  agent. 

Tests  to  confirm  toxicity  must  be  conducted  using  simulants  in  the  actual  environment.  Since  the  agent's 
toxicity  is  dependent  on  factors  such  as  droplet  size,  dispersion  density,  ground  contamination  pattern, 
and  degradation  rate,  a  simulant  that  behaves  as  the  agent  does  must  be  used  in  actual  field  testing. 
Agent  toxicity  is  determined  in  the  lab. 

24.2.2  Laser  Weapons  Testing 

Many  new  weapon  systems  are  being  designed  with  embedded  laser  range  finders  and  laser 
designators.  Because  of  the  danger  to  the  human  eye  posed  by  lasers,  the  tester  must  adhere  to  special 
safety  requirements  and  utilize  specially  configured  geographic  locations  during  T&E.  For  instance,  the 
program  seeking  to  free-play  non-eye-safe  airborne  or  ground  lasers  during  tests  involves  significant 
precautions,  such  as  the  airspace  must  be  restricted  from  overflight  during  active  testing  and  guards 
must  be  posted  to  prevent  anyone  (hikers,  bikers,  off-road  vehicles,  equestrians)  from  accidentally 
venturing  into  the  area.  A  potential  solution  to  the  safety  issue  is  to  develop  and  use  proper  procedures 
to  ensure  a  non-tactical  system  operation  that  allows  the  actual  design  hardware/software  to  be  part  of 
the  unit  under  test.  The  tester  must  ensure  that  eye-safe  lasers  produce  the  same  laser  energy  as  the 
real  laser  system.  Another  concern  of  the  laser/directed-energy  weapons  tester  is  the  accurate 
determination  of  energy  levels  and  location  on  the  target.  Measurements  of  the  energy  on  the  target 
are  usually  conducted  in  the  laboratory.  In  the  field,  video  cameras  are  often  used  to  verify  that  a  laser 
designator  did  indeed  illuminate  the  target.  Such  determinations  are  important  when  the  tester  is  trying 
to  attribute  weapon  performance  to  behavior  of  the  directed-energy  weapon,  behavior  of  the  guidance 
system,  or  some  other  factor. 
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24.3  Space  System  Testing 

From  a  historical  perspective,  space  system  acquisition  has  posed  several  unique  problems  to  the  test 
process,  especially  the  OT  process.  The  root  causes  generally  fall  into  four  categories:  limited 
quantities/high  cost,  incremental  upgrade  approach  to  acquisition,  operating  environment  (peacetime 
and  wartime),  and  test  environment.  These  root  causes  are  summarized  below. 

Limited  Quantities/High  Cost .  Space  systems  have  traditionally  involved  the  acquisition  of  relatively 
few  (historically,  less  than  20)  systems  at  extremely  high  per-unit  costs  (in  comparison  with  more 
traditional  military  systems).  The  high  per-unit  costs  are  driven  by  a  combination  of  high  transportation 
costs  (launch  to  orbit),  high  life-cycle  reliability  requirements  and  associated  costs  because  of  the  lack  of 
an  on-orbit  maintenance  capability,  and  the  high  costs  associated  with  leading  edge  technologies  that 
tend  to  be  a  part  of  spacecraft  design.  From  a  test  perspective,  this  situation  serves  to  drive  space 
system  acquisition  strategy  into  a  nonstandard  approach  addressed  below.  The  problem  is  compounded 
by  the  incremental  upgrade  approach  to  acquisition. 

Incremental  Upgrade  Approach  to  Acquisition  .  Due  to  the  limited  buy  and  high  per-unit  cost  nature  of 
spacecraft  acquisition,  these  systems  tend  to  be  procured  using  an  individual  increment  acquisition 
strategy.  Under  this  concept,  the  decision  to  deploy  is  often  made  at  the  front  end  of  the  acquisition 
cycle,  and  the  first  prototype  to  be  placed  in  orbit  becomes  the  first  operational  asset.  As  early  and 
follow-on  systems  undergo  ground  and  on-orbit  testing  (either  DT&E  or  OT&E),  discrepancies  are 
corrected  by  increment  changes  to  the  next  system  in  the  pipeline.  This  approach  to  acquisition  can 
perturb  the  test  process  as  the  tester  may  have  no  formal  milestone  decisions  to  test  toward.  The  focus 
must  change  toward  being  able  to  influence  the  design  of  (and  block  changes  to)  systems  further 
downstream  in  the  pipeline.  Because  the  first  on-orbit  asset  usually  becomes  the  first  operational  asset, 
pressure  is  created  from  the  operational  community  to  expedite  (and  sometimes  limit)  testing  so  a 
limited  operational  capability  can  be  declared  and  the  system  can  begin  fulfilling  mission  requirements. 
Once  the  asset  goes  operational,  any  use  of  it  for  testing  must  compete  with  operational  mission  needs- 
a  situation  potentially  placing  the  tester  in  a  position  of  relatively  low  priority.  Recognition  of  these 
realities  and  careful  early-on  test  planning  can  overcome  many  of  these  problems,  but  the  tester  needs 
to  be  involved  and  ready  early  in  the  life  cycle. 

Operating  Environment  (Peacetime  and  Wartime) .  Most  currently  deployed  space  systems  and  near- 
term  future  space  systems  operate  in  the  military  support  arena  such  as  tactical  warning/attack 
assessment,  communications,  navigation,  weather,  and  intelligence,  and  their  day-to-day  peacetime 
operating  environment  is  not  much  different  from  the  wartime  operating  environment  except  for 
activity  level  (i.e.,  message  throughput,  more  objects  to  track/see,  etc.).  Flistorically,  space  has  been  a 
relatively  benign  battlefield  environment  because  of  technology  limitations  in  the  capability  of  potential 
adversaries  to  reach  into  space  with  weapons.  But  this  is  no  longer  valid.  This  combination  of  support- 
type  missions  and  a  battlefield  environment  that  is  not  much  different  from  the  peacetime  environment 
has  played  a  definite  role  in  allowing  systems  to  reach  limited  operational  capability  without  as  much 
dedicated  prototype  system-level  testing  as  seen  on  other  systems.  This  situation  is  changing  with  the 
advent  of  organizations  like  the  Missile  Defense  Agency,  in  which  actual  weapons  systems  (impact 
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antisatellite  and  laser)  may  be  in  operation,  and  day-to-day  peacetime  operations  will  not  mirror  the 
anticipated  battlefield  environment  as  closely.  Likewise,  the  elevation  of  the  battlefield  into  space  and 
the  advancing  technologies  that  allow  potential  adversaries  to  reach  into  space  are  changing  the  thrust 
of  how  space  systems  need  to  be  tested  in  space. 

Test  Environment .  The  location  of  space  assets  in  remote  orbits  also  compounds  the  test  problem. 
Space  systems  do  not  have  the  ready  access  (as  with  ground  or  aircraft  systems)  to  correct  deficiencies 
identified  during  testing.  This  situation  has  driven  the  main  thrust  of  testing  into  the  prelaunch  critical 
component  testing  and  ground  simulation  (such  as  altitude  chamber)  test  environment  in  which 
discrepancies  can  be  corrected  before  the  system  becomes  inaccessible.  However,  as  mentioned 
previously,  when  space  system  missions  change  from  a  peacetime  focus  to  a  warfighting  focus  and  the 
number  of  systems  required  to  do  the  mission  increases  from  the  high  reliability/limited  number  mode 
to  a  more  traditional  fairly  large  number  buy  mode,  future  space  system  testing  could  be  expected  to 
become  more  like  the  testing  associated  with  current  ground,  sea,  and  air  systems.  From  a  test 
perspective,  this  change  could  also  create  unique  test  technology  requirements;  i.e.,  with  these  systems, 
it  will  be  necessary  to  bring  the  test  range  to  the  operating  system  as  opposed  to  bringing  the  system  to 
the  range.  Also,  because  the  space  environment  tends  to  be  visible  to  the  world  (others  can  observe  our 
tests  as  readily  as  we  can),  unique  test  operations  security  methodologies  may  be  required  to  achieve 
test  realism  without  giving  away  system  vulnerabilities. 

In  summary,  current  and  near-term  future  space  systems  have  unique  test  methodologies.  However,  in 
the  future,  space  operations  might  entail  development/deployment  of  weapon  platforms  on  orbit  with 
lower  design-life  reliability  (because  of  cost),  and  day-to-day  peacetime  operations  will  not  mirror  the 
wartime  environment.  Thus,  space  system  testing  requirements  may  begin  to  more  closely  parallel 
those  of  traditional  weapon  systems. 

24.4  Cyber  Warfare 

24.4.1  Introduction 

Cyber  warfare  refers  to  politically  motivated  hacking  to  conduct  sabotage  and  espionage.  It  is  a  form  of 
information  warfare  sometimes  seen  as  analogous  to  conventional  warfare  although  this  analogy  is 
controversial  for  both  its  accuracy  and  its  political  motivation. 

Cyber  warfare  poses  a  significant  threat  to  U.S.  military  capabilities:  Determined  cyber  foes  can  threaten 
our  global  logistics  network,  steal  our  operational  plans,  blind  our  intelligence  capabilities,  or  hinder  our 
ability  to  deliver  weapons  on  target.  The  frequency  and  sophistication  of  intrusions  into  U.S.  military 
computers,  information  systems,  and  communications  networks  have  increased  significantly. 

Dominance  across  the  full  spectrum  of  operations  within  the  cyberspace  warfighting  domain  is  essential 
if  U.S.  forces  are  to  maintain  a  strategic  advantage.  The  following  are  examples  of  cyber  warfare: 

On  October  6,  2011,  it  was  announced  that  the  Creech  Air  Force  Base  drone  and  Predator  fleets  C2  data 
stream  was  keylogged  and  resisted  all  attempts  to  reverse  the  exploitation. 
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In  September  2010,  Iran  was  attacked  by  the  Stuxnet  worm,  thought  to  specifically  target  its  Natanz 
nuclear  enrichment  facility.  The  worm  was  said  to  be  the  most  advanced  piece  of  malicious  software 
(i.e.,  malware)  ever  discovered  and  significantly  increased  the  profile  of  cyber  warfare. 

In  April  2007,  Estonia  came  under  cyber  attack  in  the  wake  of  relocation  of  the  Bronze  Soldier  of  Tallinn. 
The  largest  part  of  the  attacks  came  from  Russia  and  from  official  servers  of  the  authorities  of  Russia.  In 
the  attack,  ministries,  banks,  and  media  were  targeted. 

The  2008  cyber  attack  on  the  United  States  was  to  date  the  worst  breach  of  U.S.  military  computers  in 
history.  It  led  to  the  creation  of  U.S.  Cyber  Command  (USCYBERCOM).  It  started  when  a  universal  serial 
bus  (USB)  flash  drive  infected  by  a  foreign  intelligence  agency  was  left  in  the  parking  lot  of  a  DoD  facility 
at  a  base  in  the  Middle  East.  The  drive  contained  malicious  code  and  was  put  into  a  USB  port  from  a 
laptop  computer  that  was  attached  to  U.S.  Central  Command.  The  Pentagon  spent  nearly  14  months 
cleaning  the  worm  from  military  networks.  The  worm  had  the  ability  to  scan  computers  for  data,  open 
backdoors,  and  send  through  those  backdoors  to  a  remote  command  and  control  server. 

Critical  to  achieving  cyber  warfare  dominance  is  mission  assurance-the  ability  to  ensure  the  successful 
completion  of  military  objectives  while  operating  in  a  contested  cyber  environment  with  sophisticated 
adversaries.  The  original  information  infrastructure,  despite  security  enhancements,  was  designed  for  a 
benign  environment. 

Cyber  and  IT  security  should  be  considered  across  the  life  cycle.  The  Acquisition  Strategy  should  address 
design  and  programmatic  measures/countermeasures  to  protect  the  programs  technologies  and 
capabilities.  It  should  also  describe  the  processes  planned  to  be  used  to  identify  critical  program 
information  (CPI)  and,  if  applicable,  measures  for  the  protection  of  CPI,  including  protection  of  critical 
functionality  from  supply  chain  risk  in  accordance  with  DoDI  5200.39  (Reference  (cl))  and  DTM  09-016 
(Reference  (cm)).The  Acquisition  Strategy  should  identify  the  technical,  schedule,  cost,  and  funding 
issues  associated  with  protecting  critical  program  information  and  technologies  and  the  plans  to  resolve 
the  issues. 

MACS  and  CLs  called  out  in  Reference  (bu)  set  the  security  level  for  any  particular  system.  The  security 
level  of  the  system  determines  what  types  of  lA  controls  must  be  put  into  place  for  the  system;  this  is 
the  DIACAP.  Although  the  DIACAP  requires  lA  controls  to  be  put  into  place,  it  is  a  compliance  program 
and  not  a  test  program.  In  today's  perilous  cyber  environment,  PMs  should  be  going  beyond  the  DIACAP 
and  subjecting  their  systems  to  more  thorough  cyber  testing. 

The  following  subsections  discuss  the  concepts  of  computer  network  operations  (CNO)  including  CND, 
computer  network  exploitation  (CNE),  and  computer  network  attack  (CNA),  as  well  as  section  933  of 
Public  Law  111-383  (Reference  (cn)),  the  DoD  strategy  for  rapid  acquisition  of  tools,  applications,  and 
other  capabilities  for  cyber  warfare. 
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24.4.2  CNO 

CNO  have  become  vital  in  todays  interconnected  world.  Protecting  our  information  networks  allows  our 
forces  to  operate  effectively,  and  frequently  with  advantages,  in  today's  battlefield.  At  the  same  time, 
offensive  cyber  operations  can  be  utilized  to  deny  our  adversaries  that  same  benefit.  Additionally, 
offensive  cyber  operations  may  be  used  to  inflict  actual  damage  on  adversary  systems  or  to  gather 
information  from  their  networks.  The  following  sections  cover  some  special  concerns  that  arise  when 
testing  CNO  capabilities.. 

24.4.2.1  CND 

Definition  -  As  defined  in  Joint  Publication  3-13  (Reference  (co)),  actions  taken  through  the  use  of 
computer  networks  to  protect,  monitor,  analyze,  detect,  and  respond  to  unauthorized  activity  within 
DoD  information  systems  and  computer  networks. 

CND  testing  is  used  to  ensure  that  CND  tools  will  protect  our  networks  and  systems  from  outside 
threats.  Additional  considerations  to  address  in  CND  testing  include  the  following: 

Threat.  An  appropriate  threat  assessment  must  be  utilized  to  determine  adversary  capabilities  and  how 
those  capabilities  can  exploit  system  vulnerabilities. 

Red  Team.  Threat  information  should  be  used  by  a  team  of  red  attackers  to  fully  exercise  system  and 
network  defenses. 

Operationally  Realistic  Ranges  .  CND  testing  should  involve  operationally  realistic  ranges.  Blue, 
adversary,  and  neutral  networks  should  be  simulated  or  brought  in  to  the  test  environment  via  a  live, 
virtual,  and  constructive  setup.  The  environment  should  also  include  realistic  network  traffic  generators 
to  simulate  loads  on  bandwidth  and  server  throughput. 

CND  Service  Providers.  CND  testing  of  a  system  should  also  include  the  CND  service  provider  that 
provides  umbrella  defense  to  the  system  under  test.  This  inclusion  will  allow  the  system  to  be  tested  in 
an  environment  closer  to  that  under  which  it  will  actually  operate-part  of  the  operationally  realistic 
range. 

Instrumentation  .  Currently,  it  is  difficult  to  know  what  is  taking  place  within  our  systems  (from  a  cyber 
defense  view)  because  systems  do  not  contain  instrumentation  and  measures  that  allow  for  insight  into 
their  inner  workings.  Future  developments  should  include  data  monitoring  and  instrumentation  to 
utilize  during  CND  testing. 

24.4.2.2  CNE 

Definition  As  defined  in  Reference  (co) ,  enabling  operations  and  intelligence  collection  capabilities 
conducted  through  the  use  of  computer  networks  to  gather  data  from  target  or  adversary  or  networks. 

Utilizing  CNE,  an  entity  may  collect  data  from  an  adversary  (and  sometimes  a  friend)  for  an  extended 
period  of  time.  The  data  gathering  helps  the  entity  at  the  strategic,  operational,  and  tactical  levels  and 


286 


525  &-yR9l3&lzuSy'a  l-yt-BSTSyuDcmS 


/KI-L0J!liK 


may  lead  to  compromise  of  sensitive  information  and  introduction  of  vulnerabilities  into  the  target 
system(s).  CNE  tools  typically  consist  of  Trojan  horse  programs  and  other  associated  malware.  In 
addition,  CNE  tools  may  be  designed  to  take  full  control  of  infected  computers  and  the  associated 
domains. 

Due  to  the  potentially  destructive  nature  of  CNE,  testing  of  CNE  tools  should  ensure  that  the  tools  are 
able  to  operate  effectively  without  having  undesired  side  effects.  In  addition,  CNE  testing  should  include 
tests  to  ensure  extended  data-gathering  capabilities  (going  undetected)  as  well  as  tests  measuring  full 
control  of  infected  computers  and  domains. 

24.4.2.3  CNA 

Definition  -  As  defined  in  Reference  (co),  actions  taken  through  the  use  of  computer  networks  to 
disrupt,  deny,  degrade,  or  destroy  information  resident  in  computers  and  computer  networks,  or  the 
computers  and  networks  themselves. 

CNA  testing  is  used  to  ensure  that  CNA  tools  will  have  the  desired  effect.  Additional  considerations  to 
address  in  CNA  testing  include  the  following: 

Collateral  Effects  .  If  tools  built  to  carry  out  CNA  operations  are  not  designed  and  built  carefully,  they 
can  have  unintended  collateral  effects  when  employed.  Even  when  carefully  constructed,  these  tools 
may  have  some  level  of  collateral  effects  that  must  be  tolerated.  Thus  before  CNA  tools  can  be 
employed,  they  must  be  thoroughly  tested  so  that  their  collateral  effects  are  understood. 

Detection  .  CNA  tool  testing  needs  to  examine  the  likelihood  of  detection  by  adversaries.  Detection  can 
lead  to  an  adversary  shutting  down  an  avenue  of  attack.  Potentially  worse,  if  an  adversary  is  able  to 
detect  and  trace  an  attack  back  to  its  source,  its  attribution  could  lead  to  unwanted  political 
consequences. 

Closed  and  Operationally  Realistic  Ranges  .  Testing  for  CNA  tools  and  capabilities  should  take  place  on 
closed  ranges  where  the  potential  spread  of  adverse  effects  can  be  contained.  In  addition  to  being 
isolated,  these  ranges  should  provide  operationally  realistic  environments  that  simulate  blue,  adversary, 
and  neutral  networks.  Unfortunately,  although  fairly  accurate  models  of  adversary  networks  can  be 
constructed,  the  models  will  never  be  perfect.  Therefore,  a  full  and  complete  CNA  OT  will  likely  not  be 
possible,  as  some  details  of  the  adversaries  systems  will  become  known  only  during  an  operational 
effort. 

DoD  Instruction  0-3600.03  (Reference  (cp)) .  Technical  Assurance  Standard  (TAS)  for  Computer 
Network  Attack  (CNA)  Capabilities  establishes  guidelines  and  standards  for  CNA  tools. 

24.4.3  Rapid  Acquisition  of  Tools,  Applications,  and  Other  Capabilities 

Section  933  of  Reference  (cn)  directs  the  DoD  to  provide  a  strategy  for  the  rapid  acquisition  of  tools, 
applications,  and  other  capabilities  for  cyber  warfare  for  USCYBERCOM  and  the  cyber  operations 
components  of  the  Military  Departments.  The  USD(AT&L)  worked  with  the  DoD  cyber  community  to 
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develop  a  common  framework  for  the  Services  and  Defense  Agencies  to  acquire  cyberspace  operations 
capabilities. 

The  framework  addresses  requirements,  acquisition,  testing,  and  governance.  To  execute  this  new 
framework,  the  traditional  defense  acquisition  framework  will  be  aggressively  streamlined  to 
accommodate  cyberspace  operations  quick  reaction  timelines,  while  at  the  same  time  managing  risk. 
There  are  four  cyber  capability  need  timelines:  A  less  than  30  days;  B  30  days  to  9  months;  C  9  to  18 
months;  D  greater  than  18  months  (likely  to  follow  traditional  acquisition  processes). 

An  evaluation  of  operational  risk  to  mission  success  is  central  to  testing  in  all  four  cyberspace  operations 
acquisition  processes  and  will  be  the  primary  driver  in  determining  the  degree  of  testing  appropriate  for 
each  capability.  Whichever  of  the  four  processes  is  followed,  an  integrated  team  of  testers,  certifiers, 
and  users  must  help  develop  the  requirements,  identify  the  significant  operational  risks,  and  work  to 
address  those  risks  through  T&E.  T&E  of  cyberspace  operations  capabilities  must  support  risk 
management  in  any  of  the  four  acquisition  paths  by  providing  decision  makers  with  credible,  relevant, 
and  efficient  estimates  of  system  and  operational  performance.  T&E  processes  and  products  must  be 
tailored  to  program  need,  risk,  and  risk  tolerance;  be  fully  integrated  into  capability  development 
processes;  leverage  testing  as  a  service;  and  efficiently  synthesize  developmental,  operational,  and 
specialty  T&E  perspectives  to  generate  data  for  independent  evaluation. 

24.4.4  Cyber  Warfare  in  the  United  States 

The  DoD  sees  the  use  of  computers  and  the  Internet  to  conduct  warfare  in  cyberspace  as  a  threat  to 
national  security.  According  to  the  U.S.  Joint  Forces  Command,  cyberspace  technology  is  emerging  as  an 
instrument  of  power  in  societies,  and  is  becoming  more  available  to  a  country's  opponents,  who  may 
use  it  to  attack,  degrade,  and  disrupt  communications  and  the  flow  of  information.  With  low  barriers  to 
entry,  coupled  with  the  anonymous  nature  of  activities  in  cyberspace,  the  list  of  potential  adversaries  is 
broad.  Furthermore,  the  globe-spanning  range  of  cyberspace  and  its  disregard  for  national  borders  will 
challenge  legal  systems  and  complicate  a  nation's  ability  to  deter  threats  and  respond  to  contingencies. 

24.5  Operations  Security  and  T&E 

OPSEC  issues  must  be  considered  in  all  test  planning.  Security  regulations  and  contracting  documents 
require  the  protection  of  sensitive  design  information  and  test  data  throughout  the  acquisition  cycle  by: 

•  Protecting  sensitive  technology. 

•  Eliminating  non-secure  transmittal  data  on  and  from  test  ranges. 

•  Providing  secure  communications  linking  DoD  agencies  to  each  other  and  to  their  contractors. 

Such  protection  is  obviously  costly  and  will  require  additional  planning  time,  test  resources,  and  test 
constraints.  The  test  planner  must  determine  all  possible  ways  in  which  the  system  could  be  susceptible 
to  hostile  exploitation  during  testing.  For  example,  announcement  of  test  schedule  and  location  could 
allow  monitoring  by  unauthorized  persons.  Knowledge  of  the  locations  of  systems  and  instrumentation 
or  test  concepts  could  reveal  classified  system  capabilities  or  military  concepts.  Compilations  of 
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unclassified  data  could,  as  a  whole,  reveal  classified  information  as  could  surveillance  (electronic  or 
photographic)  of  test  activities  or  interception  of  unencrypted  transmissions. 

Routine  reviews  of  the  system  under  test  security  classification  guide  should  be  standard  practice  of  the 
test  team.  Specific  issues  of  interest  should  be  identified  before  each  test  event  that  has  potential 
exposure  risk.  Some  new  acquisitions  may  not  have  complete  classification  guidance,  requiring  test 
teams  to  be  proactive  in  seeking  guidance  for  new  technologies  and  unexplored  CONORS. 

24.6  Joint  Capability  Technology  Demonstrations 

Systems  with  potential  utility  for  the  user  and  having  relatively  mature  technology  may  be  evaluated  by 
a  user  in  an  operational  field  environment.  These  programs  are  not  an  acquisition  program  and 
therefore  are  not  subject  to  the  normal  acquisition  T&E  processes.  A  favorable  evaluation  may  result  in 
the  decision  to  acquire  additional  systems  for  Service  use,  bypassing  a  number  of  the  normal  acquisition 
phases.  The  Services  have  been  using  their  OTAs  to  assist  the  field  commanders  in  structuring  an 
evaluation  process  that  would  provide  the  documented  data  necessary  for  an  informed  acquisition 
decision. 

24.7  Chemical  and  Biological  Agent  Testing 

The  Chemical  and  Biological  Defense  Program  uses  chemical  and  biological  agents  in  controlled  amounts 
(and  in  compliance  with  the  relevant  treaties)  to  test  the  efficacy  of  protection,  decontamination, 
detection,  medical,  modeling,  and  warning  systems.  The  testing  of  chemical  and  biological  defense 
equipment  and  medical  countermeasures  systems  poses  unique  problems  because  the  tester  cannot 
perform  actual  open-air  field  testing.  In  addition  to  obvious  health  and  safety  factors,  test  issues  that 
the  chemical  and  biological  system  tester  must  address  include  the  following: 

•  All  possible  chemical  reactions  due  to  variations  such  as  moisture,  temperature,  pressure,  and 
contamination. 

•  Physical  behavior  of  the  chemical;  i.e.,  droplet  size,  dispersion  density,  and  ground 
contamination  pattern  when  used  operationally. 

•  Toxicity  of  the  chemical;  i.e.,  lethality  and  duration  of  contamination  when  used  operationally. 

•  Safety  of  the  chemical  weapon  during  storage,  handling,  and  delivery;  adherence  to  chemical 
and  biological  surety  regulations. 

•  Decontamination  processes. 

Addressing  all  of  these  issues  requires  a  variety  of  techniques  and  test  fixtures  ranging  from  small-scale, 
material-level  laboratory  tests  in  which  chemical  and  biological  agents  are  employed,  to  large  chamber 
and  open-air  field  testing  of  systems.  Cost  and  safety  concerns  may  require  the  use  of  simulants,  which 
are  substances  that  replicate  the  physical  or  chemical  properties  of  the  tested  agent  but  possess 
reduced  human  and  environmental  toxicity. 

It  should  be  noted,  however,  that  simulant-agent  correlation  is  very  difficult  to  establish  and  results  in 
severe  limitations  in  test  results,  particularly  in  determining  the  efficacy  of  detection,  protection,  and 
decontamination  systems.  In  medical  systems,  correlation  of  simulant  to  agent  properties  is  not  a 
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reliable  method  for  establishing  the  performance  of  most  medical  countermeasures.  Studies  of  agent 
chemistry  are  performed  in  toxic-chamber  tests  using  the  agents  to  establish  physical  properties, 
solubility,  and  interaction  with  common  battlefield  contaminants.  Factors  such  as  humidity, 
temperature,  pressure,  and  agent  purity  must  be  controlled  to  assess  the  agent's  behavior. 

The  ability  to  predict  operational  outcomes  upon  exposure  to  chemical  and  biological  warfare  agents  is 
further  complicated  by  the  limited  available  human  toxicity  data.  Chemical  surety  in  U.S.  laboratories 
and  test  ranges  is  governed  by  AR  50-6  (Reference  (cq)),  and  biological  surety  is  governed  by  AR  50-1 
(Reference  (cr)). 

24.8  Summary 

All  weapon  systems  tests  are  limited  to  some  degree,  but  certain  systems  face  major  limitations  that 
could  preclude  a  comprehensive  and  realistic  evaluation.  The  test  planners  of  these  special  systems 
must  allow  additional  planning  time,  budget  for  extra  test  resources,  and  devise  alternative  test 
strategies  to  work  around  testing  limitations  caused  by  such  factors  as  security  restrictions,  resource 
availability,  environmental  safety  factors,  and  nonstandard  acquisition  strategies. 
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A 

A  A 

AAE 

ACAT 

ACE 

ACTD 

ADM 

AEC 

AFMC 

AFOTEC 

AFSPC 

AF/TE 

A, 

AIS 

AMC 

Aq 

AoA 

AOTR 

APB 

AR 

ARE 

ASA(ALT) 

ASAF(A) 

ASD(R&E) 

ASN(RD&A) 

ASR 


Appendix  A:  Acronyms  and  Abbreviations 


Achieved  Availability 

Army  Acquisition  Executive 

Acquisition  Category 

Army  Corps  of  Engineers 

Advanced  Concept  Technology  Demonstration 

Acquisition  Decision  Memorandum 

Army  Evaluation  Center 

Air  Force  Materiel  Command 

Air  Force  Operational  Test  and  Evaluation  Center 

Air  Force  Space  Command 

Air  Force  Test  and  Evaluation 

Inherent  Availability 

Automated  Information  System 

Army  Materiel  Command 

Operational  Availability 

Analysis  of  Alternatives 

Assessment  of  Operational  Test  Readiness 

Acquisition  Program  Baseline 

Army  Regulation 

Army  Research  Laboratory 

Assistant  Secretary  of  the  Army  (Acquisition,  Logistics  and  Technology) 

Assistant  Secretary  of  Defense  for  Research  and  Engineering 

Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acquisition 

Alternative  Systems  Review 
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AT 

ATD 

ATE 

ATEC 

ATO 

AV 

B 

BCD 

BCL 

BEA 

BIT 

BITE 

BLRIP 

C 

C&A 

C2 

C4 

C4I 

CAD/CAM 

CAE 

CARD 

CBA 

CBM 

CDD 

CDR 

CDRL 

CEP 


Anti-Tamper 

Advanced  Technology  Development  (or  Demonstration) 

Automated  Test  Equipment 

Army  Test  and  Evaluation  Command  (Army) 

Authority  to  Operate  (DIACAP) 

All  Viewpoint 

Business  Capability  Definition 
Business  Capability  Lifecycle 
Business  Enterprise  Architecture 
Built-In  Test 
Built-in  Test  Equipment 
Beyond  Low  Rate  Initial  Production 

Certification  and  Accreditation 
Command  and  Control 

Command,  Control,  Communications,  and  Computers 
Command,  Control,  Communications,  Computers  and  Intelligence 

Component  Acquisition  Executive;  Computer-Aided  Engineering 

Cost  Analysis  Requirements  Description 

Cost-Benefit  Analysis 

Chemical,  Biological,  and  Medical 

Capability  Development  Document 

Critical  Design  Review 

Contract  Data  Requirements  List 

Circular  Error  Probable 
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CG 

Commanding  General 

Cl 

Commercial  Item;  Configuration  Item 

CJCSI 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

CL 

Confidentiality  Level 

CM 

Configuration  Management 

CNA 

Computer  Network  Attack 

CND 

Computer  Network  Defense 

CNE 

Computer  Network  Exploitation 

CNDSP 

Computer  Defense  Service  Provider 

CNO 

Chief  of  Naval  Operations 

CNO 

Computer  Network  Operations 

CNSSI 

Committee  on  National  Security  Systems  Instruction 

COCOM 

Combatant  Command 

COI 

Critical  Operational  Issues  and  Criteria 

COIC 

Computer  Network  exploitation 

COMOPTEVFOR 

Commander,  Operational  Test  and  Evaluation  Force  (Navy) 

CONOPS 

Concept  of  Operations 

COOP 

Continuity  of  Operations 

CoP 

Community  of  Practice 

COTS 

Commercial  Off-The-Shelf 

CPD 

Capability  Production  Document 

CPI 

Critical  Program  Information 

CPU 

Central  Processing  Unit 

CRTC 

Cold  Regions  Test  Center 

CSAF 

Chief  of  Staff  of  the  Air  Force 

CSCI 

Computer  Software  Configuration  Item  (also  called  SI  (Software  Item)) 

CSI 

Critical  Safety  Item 
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CTEIP 

CTO 

CTP 

CV 

CY 

D 

DAB 

DAG 

DAE 

DAES 

DAS 

DASD(DT&E) 

DAU 

DBS 

DBSMC 

DCAPE 

DCMA 

DIACAP 

DISA 

DIV 

DLT 

DMMPO 

DoD 

DoDAF 

DoDD 

DoDI 

DOE 


Central  Test  and  Evaluation  Investment  Program 
Comparative  Technology  Office 
Critical  Technical  Parameter 
Capability  Viewpoint 
Calendar  Year 

Defense  Acquisition  Board 
Defense  Acquisition  Guidebook 
Defense  Acquisition  Executive 
Defense  Acquisition  Executive  Summary 
Director  of  the  Army  Staff 

Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and  Evaluation 
Defense  Acquisition  University 
Defense  Business  System 

Defense  Business  Systems  Management  Committee 
Director  of  Cost  Assessment  and  Program  Evaluation 
Defense  Contract  Management  Agency 

DoD  Information  Assurance  Certification  and  Accreditation  Process 
Defense  Information  Systems  Agency 
Data  and  Information  Viewpoint 
Design  Limit  Test 

Defense  Medical  Materiel  Program  Office 

Department  of  Defense 

DoD  Architecture  Framework 

DoD  Directive 

DoD  Instruction 

Design  of  Experiments 
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DOT&E 

DOTMLPF 

DFARS 

DIACAP 

DID 

DISA 

DLT 

DoDAF 

DoDD 

DoDI 

DOE 

DON 

DOT&E 

DOTMLPF 

DR 

DT 

DT&E 

DTIC 

DTM 

DTRA 

DUSA-TE 

E 

E3 

ECM 

EDM 

EIS 


Director  of  Operational  Test  and  Evaluation 

(Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel,  and 
Facilities)  Change  Recommendation 

Defense  Federal  Acquisition  Regulation  Supplement 

Department  of  Defense  Information  Assurance  Certification  and  Accreditation  Process 

Data  Item  Description 

Defense  Information  Systems  Agency 

Design  Limit  Test 

Department  of  Defense  Architecture  Framework 
Department  of  Defense  Directive 
Department  of  Defense  Instruction 
Design  of  Experiments 
Department  of  the  Navy 

Director  of  Operational  Test  and  Evaluation  (Office  of  the  Secretary  of  Defense) 

Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel,  and 
Facilities  (DoD) 

Deficiency  Reporting 

Developmental  Test;  Developmental  Testing 
Developmental  Test  and  Evaluation 
Defense  Technical  Information  Center 
Directive-Type  Memorandum 
Defense  Threat  Reduction  Agency 

Deputy  Under  Secretary  of  the  Army  for  Test  and  Evaluation 

Electromagnetic  Environmental  Effects  Evolutionary  Acquisition;  Executive  Agent 
Electronic  Countermeasures 
Engineering  Development  Model 
Environmental  Impact  Statement 
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EMC 

EMD 

EMI 

EMP 

EOA 

EPG 

ERP 

ESOH 

ESS 

EVM 

EW 

F 

FAT 

FCA 

FCT 

FDA 

FDD 

FDE 

FDT/E 

FMECA 

FMO 

FOC 

FORSCOM 

FOT&E 

FRP 

FRPDR 

FRR 


Electromagnetic  Compatibility 

Engineering  and  Manufacturing  Development  (phase  of  the  Defense  Acquisition 
Management  System) 

Electromagnetic  Interference 

Electromagnetic  Pulse 

Early  Operational  Assessment 

Electronic  Proving  Ground 

Enterprise  Resource  Planning 

Environment,  Safety,  and  Occupational  Flealth 

Environmental  Stress  Screening 

Earned  Value  Management 

Electronic  Warfare 

First  Article  Testing 

Functional  Configuration  Audit 

Foreign  Comparative  Testing 

Food  and  Drug  Administration 

Full  Deployment  Decision 

Force  Development  Evaluation 

Force  Development  Tests  and/or  Experimentation 

Failure  Mode,  Effects,  and  Criticality  Analysis 

Frequency  Management  Office 

Full  Operational  Capability 

Follow-on  Operational  Test  and  Evaluation 

Follow-on  Operational  Test  and  Evaluation 

Full-Rate  Production 

Full-Rate  Production  Decision  Review 

Flight  Readiness  Review 
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FUSL 

FWE 

FY 

FYDP 

FYTP 

G 

GFE 

GIG 

GOTS 

H 

HERF 

HERO 

HERP 

HI 

HNA 

HPT 

HQ 

HQDA 

HSI 

HSIPP 

HWIL 

I 

lA 

lAW 

IBR 

ICBM 

ICD 


Full-Up  System-Level 
Foreign  Weapons  Evaluation 
Fiscal  Year 

Future  Years  Defense  Program 
Five-Year  Test  Program 

Government-Furnished  Equipment 
Global  Information  Grid 
Government  Off-The-Shelf 

Hazards  of  Electromagnetic  Radiation  to  Fuel 

Hazards  of  Electromagnetic  Radiation  to  Ordnance 

Hazards  of  Electromagnetic  Radiation  to  Personnel 

Hardware  Item 

Host-Nation  Approval 

High-performance  Team 

Headquarters 

Headquarters,  Department  of  the  Army 
Human  Systems  Integration 
Human  Systems  Integration  Program  Plan 
Hardware-in-the-Loop 

Information  Assurance 
In  Accordance  With 
Integrated  Baseline  Review 
Intercontinental  Ballistic  Missile 
Initial  Capabilities  Document 
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ID 

lER 

ILS 

IM 

IMO 

INCOSE 

INSCOM 

IOC 

lOT&E 

IPS 

IPT 

IRB 

IRS 

ISD 

ISP 

ISR 

ISTF 

IT 

ITOP 

ITR 

ITT 

IV&V 

J 

J-8 

JCIDS 

JEET 

JITC 


Information  Evaluation  Report 
Integrated  Logistics  Support 
Investment  Management 
Instrumentation  Management  Office 
International  Council  on  Systems  Engineering 
Intelligence  and  Security  Command 
Initial  Operational  Capability 
Initial  Operational  Test  and  Evaluation 
integrated  product  support 
Integrated  Product  Team 
Investment  Review  Board 
Interface  Requirement  Specification 

Integrated  System  Design  (effort  of  the  Engineering  Manufacturing  Development 
phase) 

Information  Support  Plan;  Internet  Service  Provider 

In-Service  Review 

Installed  System  Test  Facility 

Information  Technology 

International  Test  Operations  Procedures 

Initial  Technical  Review 

integrated  test  team 

Independent  Verification  and  Validation 

Joint  Staff  Directorate  for  Force  Structure,  Resource,  and  Assessment 

Joint  Capabilities  Integration  and  Development  System 

Joint  E3  Evaluation  Tool 

Joint  Interoperability  Test  Command 
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JLF 

JROC 

JT&E 

JTCG 

K 

KLP 

KPP 

KSA 

L 

LCC 
LCSP 
LFT 
LFT&E 
LOG  Demo 
LOG  T&E 
LRIP 
LSA 

M 

MAC 

MAIS 

MAJCOM 

MCEB 

MCOTEA 

MCSC 

MDA 

MDAP 

MDD 


Joint  Live  Fire 

Joint  Requirements  Oversight  Council 

Joint  Test  and  Evaluation 

Joint  Technical  Coordinating  Group 

Key  Leadership  Position 
Key  Performance  Parameter 
Key  System  Attribute 

Life  Cycle  Cost 

Life  Cycle  Sustainment  Plan 

Live-fire  Testing 

Live  Fire  Test  and  Evaluation 

Logistics  Demonstration 

Logistics  Support  Test  and  Evaluation 

Low-Rate  Initial  Production 

Logistics  Support  Analysis  (Obsolete) 

Mission  Assurance  Category 
Major  Automated  Information  System 
Major  Command 

Military  Communications-Electronics  Board 
Marine  Corps  Operational  Test  and  Evaluation  Activity 
Marine  Corps  Systems  Command 
Milestone  Decision  Authority;  Missile  Defense  Agency 
Major  Defense  Acquisition  Program 

Materiel  Development  Decision  (of  the  Defense  Acquisition  Management  System) 
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MEDCOM 

MF 

MIL-HDBK 

MIL-STD 

MLDT 

MMT 

MOA 

MOE 

MOP 

MOS 

MOT&E 

MOU 

MRTFB 

M&S 

MS  or  M/S 

MSA 

MTBF 

MTBM 

MTBOMF 

MTTR 

N 

NASA 

NATO 

NAVAIR 

NAVSEA 

NBC 

NCE 


Medical  Command 
Measurement  facility 
Military  Flandbook 
Military  Standard 
Mean  Logistics  Delay  Time 

Manufacturing  Methods  Technology;  Mean  Maintenance  Time 

Memorandum  of  Agreement 

Measure  of  Effectiveness 

Measure  of  Performance 

Measure  of  Suitability 

Multi-Service  Operational  Test  and  Evaluation 
Memorandum  of  Understanding 
Major  Range  and  Test  Facility  Base 
Modeling  and  Simulation 
Milestone 

Materiel  Solution  Analysis  (phase  of  the  Defense  Acquisition  Management  System) 

Mean  Time  Between  Failure 

Mean  Time  Between  Maintenance 

Mean  Time  Between  Operational  Mission  Failures 

Mean  Time  to  Repair 

National  Aeronautics  and  Space  Administration 
North  Atlantic  Treaty  Organization 
Naval  Air  Systems  Command 
Naval  Sea  Systems  Command 
Nuclear,  Biological,  and  Chemical 
Net-centric  Environment 
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NCT  NATO  Comparative  Test 

NDAA  National  Defense  Authorization  Act 

NDI  Non-Developmental  Item 

NEPA  National  Environmental  Policy  Act 

NH&S  Nuclear  Hardness  and  Survivability 

NIST  National  Institute  of  Standards  and  Technology 

NR-KPP  Net-Ready  Key  Performance  Parameter 

NSS  National  Security  Strategy;  National  Security  System 

0 

OA  Operational  Assessment 

OAR  Open  Air  Range 

OIPT  Overarching  Integrated  Product  Team 

O&M  Operation  and  Maintenance 

0MB  Office  of  Management  and  Budget 

OPEVAL  Operational  Evaluation  (Navy) 

OPSEC  Operations  Security 

OPTEVFOR  Operational  Test  and  Evaluation  Force  (Navy) 

O&S  Operations  and  Support  (phase  of  the  Defense  Acquisition  Management  Framework); 

also  a  life  cycle  cost  category 

OSD  Office  of  the  Secretary  of  Defense 

OT  Operational  Testing 

OTA  Operational  Test  Agency 

OTC  Operational  Test  Command 

OTD  Operational  Test  Director 

OT&E  Operational  Test  and  Evaluation 

OTP  Operational  Test  Plan 

OTRR  Operational  Test  Readiness  Review 

OV  Operational  Viewpoint 
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P 

P3I 

PAT&E 

PCA 

PCDRA 

PCO 

PCR 

P&D 

PDR 

PE 

PEO 

PEO  STRI 

PESHE 

PHS&T 

PI 

PM 

PMB 

PMITTS 

PMO 

POA&M 

PPBE 

PPP 

PQT 

PRAT 

PRR 

PSM 

P-Static 


Preplanned  Product  Improvement 
Production  Acceptance  Test  and  Evaluation 
Physical  Configuration  Audit 
Post-Critical  Design  Review  Assessment 
Primary  Contracting  Officer 
Physical  Configuration  Review 

Production  and  Deployment  (phase  of  the  Defense  Acquisition  Management  System) 
Post-Deployment  Review;  Preliminary  Design  Review;  Program  Deviation  Report 
Program  Element 
Program  Executive  Officer 

Program  Executive  Office  for  Simulations,  Training,  and  Instrumentation 
Programmatic  Environment,  Safety  and  Occupational  Health  Evaluation 
Packaging,  Handling,  Storage,  and  Transportation 
Product  Improvement 

Product  Manager;  Program  Manager;  Project  Manager 
Performance  Measurement  Baseline 

Project  Manager  for  Instrumentation,  Targets,  and  Threat  Simulators 

Program  Management  Office 

Plan  of  Action  and  Milestones 

Planning,  Programming,  Budgeting,  and  Execution 

Program  Protection  Plan 

Production  Qualification  Test 

Production  Reliability  Acceptance  Test 

Production  Readiness  Review 

Product  Support  Manager 

Precipitation  Static 
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PV 

Q 

QA 

QASP 

QOT&E 

QRTWG 

R 

R&D 

RAM 

RDA 

RDECOM 

RDT 

RDT&E 

RF 

RFP 

RGC 

RGT 

R&M 

RM 

RTM 

RTO 

RTS 

S 

SAE 

SAF(AQ) 

SAR 

SCA 


Project  Viewpoint 

Quality  Assurance 
Quality  Assurance  Surveillance  Plan 
Qualification  Qperational  Test  and  Evaluation 
Quick  Reaction  Test  Working  Group 

Research  and  Development 

Reliability,  Availability,  and  Maintainability 

Research,  Development,  and  Acquisition 

Research,  Development,  and  Engineering  Command 

Reliability  Development  Testing 

Research,  Development,  Test,  and  Evaluation 

Radio  Frequency 

Request  for  Proposal 

Reliability  Growth  Curve 

Reliability  Growth  Testing 

Reliability  and  Maintainability 

Resource  Manager 

Requirements  Traceability  Matrix 

Responsible  Test  Qrganization 

Reagan  Test  Site 

Service  Acquisition  Executive 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Safety  Assessment  Report;  Selected  Acquisition  Report;  Special  Access  Required 
Security  Control  Assessor 
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SCI 

SC&MPD 

SE 

SecDef 

SECNAV 

SECNAVINST 

SEP 

SEP 

SETR 

SFR 

SI 

SIL 

SLAD 

SM 

SMDC 

SME 

SOW 

SP 

SPAWAR 

SPO 

SQT 

SRR 

SRS 

SS 

SSA 

SSP 


Software  Configuration  Item 

System  Capability  and  Manufacturing  Process  Demonstration  (effort  of  the 
Engineering  and  Manufacturing  Development  phase) 

Support  Equipment;  Systems  Engineering 

Secretary  of  Defense 

Secretary  of  the  Navy 

Secretary  of  the  Navy  Instruction 

Systems  Engineering  Plan;  System(s)  Engineering  Process 
System  Evaluation  Plan 
Systems  Evaluation  Plan 
System  Functional  Review 

Software  Item  (also  called  CSCI  (Computer  Software  Configuration  Item));  Special 
Intelligence 

System  Integration  Laboratory 
Survivability/Lethality  Analysis  Directorate 
Spectrum  Management 
Space  and  Missile  Defense  Command  (Army) 

Subject  Matter  Expert 
Statement  of  Work 
Special  Publication 

Space  and  Naval  Warfare  Systems  Command 
System  Program/Project  Office  (Air  Force) 

Software  Qualification  test 
System  Requirements  Review 
Software  Requirement  Specification 
Software  Specification  Review 
Spectrum  Supportability  Assessment 
Simulation  Support  Plan 
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SSR 

STA 

STAR 

STAT 

StdV 

SVR 

T 

TAAF 

TAFT 

TAP 

TD 

TDS 

T&E 

TECHEVAL 

TEMG 

TEMP 

TES 

TM 

TMO 

TOC 

TPM 

TRA 

TRADOC 

TREE 

TRIMS 

TRL 

TRMC 


Software  Specification  Review 
System  Threat  Assessment 
System  Threat  Assessment  Report 
Scientific  Test  and  Analysis  Techniques 
Standards  Viewpoint 
System  Verification  Review 

Test,  Analyze,  and  Fix 
Test,  Analyze,  Fix,  and  Test 

Technology  Development  (phase  of  the  Defense  Acquisition  Management  System) 
Technology  Development  Strategy 
Test  and  Evaluation 
Technical  Evaluation  (Navy) 

Test  and  Evaluation  Management  Guide 

Test  and  Evaluation  Master  Plan 

Test  and  Evaluation  Strategy 

Test  Manager 

Target  Management  Office 

Total  Ownership  Cost 

Technical  Performance  Measurement 

Technology  Readiness  Assessment 

Training  and  Doctrine  Command  (Army) 

Transient  Radiation  Effects  on  Electronics 
Test  Resource  Information  Management  System 
Technology  Readiness  Level 
Test  Resource  Management  Center 


305 


525  &-yR9l3&lzuSya  l-yt-B^SyuDcms 


TRP 

TRR 

TRTC 

TSARC 

TSG 

TSMO 

TTP 


Test  Resource  Plan 

Test  Readiness  Review 

Tropical  Regions  Test  Center 

Test  Schedule  and  Review  Committee 

The  Surgeon  General 

Threat  Systems  Management  Office 

Tactics,  Techniques  and  Procedures 


UAV 

USASOC 

U.S. 

USAF 

USB 

U.S.C. 

USCYBERCOM 

USD(AT&L) 

USD(C)/CFO 

USSOCOM 

V 


Unmanned  Aerial  Vehicle 

U.S.  Army  Special  Operations  Command 

United  States 

U.S.  Air  Force 

Universal  Serial  Bus 

United  States  Code 

U.S.  Cyber  Command 

Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
Under  Secretary  of  Defense  (Comptroller)/Chief  Financial  Officer 
U.S.  Special  Operations  Command 


VCSA  Vice  Chief  of  Staff  (Army) 

V&V  Verification  and  Validation 

VV&A  Verification,  Validation  and  Accreditation 

W 


WBS  Work  Breakdown  Structure 

WDTC  Western  Desert  Test  Center 

WIPT  Working-Level  Integrated  Product  Team 
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